Constructopaedia

Just finished arranging my travel to C++ Connections. I’m speaking on our Security work in the Standard C++ Library. I’ll be in Vegas from 7^th-11^th. Hopefully there will be a chance for some of us to meet up and chat about Libraries futures.

Martyn

One of the changes introduced by Visual C++ 2005 is a change in how we deploy the Visual C++ Runtime Libraries (CRT, MFC, ATL). Deployment of the runtimes has been a complex and controversial question across many versions. What we’ve attempted to do in this version of the product is both simplify our plan and make it more robust in the face of potential security situations. Explaining this is going to take a while, so I’m going to try to spend a few blog posts on this theme.

Let me start with a little history. This story explains some of the earliest, and most powerful motivation for changing our deployment model.

Static Linking

Microsoft C and later Visual C++ has included libraries since the very first version of the product, as does every C and C++ product. These libraries were typically static-link binaries, consisting of compiled .objs (bound into .libs) that you link to your own application. The libraries do everything from the mundane (such as running initialization code) to the fundamental (implementing printf so that you can say “hello world”). Because the libraries were static-linked into your program by the linker, they became inescapably part of your program.

This static linking had very clear implications for what Microsoft call servicing – the process of shipping updates to existing products. A product like Microsoft Office has a clear Servicing model. Customers buy Microsoft Office from their local software store, and if they later need an update to Office, they contact Microsoft to get that update.

Servicing a development tool is more complex. A customer goes and buys LovellSoft Construction Toy Organizer 1.0 from my local store. They find a bug in the software, and contact LovellSoft. LovellSoft, in turn, determines that the bug was caused by a problem with a piece of Visual C++. They contact Microsoft. Microsoft ships LovellSoft an updated version of the Visual C++ Libraries with a fix for their problem. They then use this update to build an updated copy of Construction Toy Organizer, which they ship back to their customer.

You can see how Microsoft has no direct relationship with LovellSoft’s customer. If the Construction Toy Organizer 1.0 user contacted Microsoft directly, there is no way we could have helped, even if we had wanted to, because our code has been linked in with code supplied by LovellSoft.

As you can see, static linking can present a real servicing problem if a security problem is found in static linked code. This leads to a recommendation

            Avoid static-linking library code into your application wherever possible.

When security problems have been found in static linked code in the past (such as this one), the community has had to wait until many vendors who used the library rebuilt versions of their product. This can take a long time.

Dynamic Linking

For some time, we have also shipped our runtime libraries as a dynamic-link library (DLL). DLLs bring with them several benefits (small binaries, less disk space, reduced code duplication), but also creates some new problems (installation location, versioning) – collectively known as “DLL Hell” in some circles.

DLLs make our servicing obligations more ambiguous. When you ship a DLL that you got from Microsoft, it’s technically possible for LovellSoft’s customer to come directly to Microsoft and get a new version of the DLL with the fix they need. Note that I say “technically”, because in practice, we’ve kept the servicing relationship for our library DLLs the same as the relationship for our static linked code. We provide updated DLLs to software developers, who then deploy those DLLs to their customers.

Windows

Servicing of one version of our libraries (msvcrt.dll and mfc42.dll) is even more complex because the operating system took a hard dependency on the library, meaning that they ship and service the DLL. This means that if LovellSoft are using VC6 to build their application, and they find a bug that they need a fix for, they’d need to get a fix from Visual Studio (so that they could redistribute msvcrt.dll to Windows 95 and Windows NT4), and get operating system patches for whichever of Windows 2000, Windows XP and Windows Server 2003 that their application targets. Complicated.

Of course, this problem is now purely hypothetical, because Visual C++ 6 (from Visual Studio 98) has been unsupported since the end of last month.

Installation Location Policy

The nub of DLL hell problems is a versioning one. Does everyone on the system get the same version of a DLL? Can a rogue application installer ‘roll back’ to an older, broken DLL? If a new application installs a broken version of a DLL, can other applications opt-out?

Our traditional motivation around DLLs had been “sharing” disk and memory pages, so we focused on installing DLLs like msvcr40.dll and msvcr20.dll to System32. But this strategy created the DLL hell problem, as installation of these DLLs frequently broke other programs.

Up to VC6, our advice was – install your CRT DLL in system32 at setup time, assuming it is newer, and reference count the installation. A side effect of this policy is that, in an emergency, we do have a way to update msvcrt.dll. Since it’s required to be installed in a central location, and since the Windows loader prefers loading from that location over most others, we have the ability to deploy fixes directly into system32. This isn’t as robust a plan as you might think, but it was an avenue available to us.

During the time of VC6, Microsoft started to get serious about addressing DLL Hell, and as a result for VC7, our advice was the opposite – install your CRT DLL to your application’s EXE directory and NOT to system32.

This has the advantage that one installation won’t interfere with another. It also helps create software that can be installed by non-administrators, run from networks or ‘xcopy-deployed’ – three things that we were getting much more interested in.

However, it also has the disadvantage that with the VC7 libraries, we have no way of servicing these DLLs centrally.

The Need for Central Servicing

As described above, we normally leave servicing of library DLLs to software vendors. They have the best knowledge of their customers needs, and so have best judgment on when and how to deploy updates to their software.

But, some events transcend this kind of policy. When a time-critical security problem is found in a redistributable component [such as an active worm on the Internet], Microsoft can’t just inform software developers of the problem and wait for them to update their products. Customers and partners will expect us to provide central updates to resolve these kinds of critical issues via http://windowsupdate.com.

The event linked above happened just after I took over leadership of the C++ Libraries team, and made me fundamentally rethink my assumptions about how we should ship our binaries.

A Solution

Luckily, in 1999 a team in Windows started work on creating solutions for exactly these kinds of problems. Their efforts resulted in the addition of manifest support to Microsoft Windows XP, and provided the foundation for us to solve the servicing problem described above, as well as several others that faced us. The solution means that most Visual C++-build DLLs and EXEs have a manifest, but also means that, if an emergency happens, we’ll be much better prepared than we were in previous versions.

In a future entry I’ll describe how this solution works, and other benefits it has, and provide some tips and tricks for problems you might hit.

In the mean time, write and tell me what you think.

Martyn

Last time I talked about how we used template overloads to help automatically transform safe calls to strcpy into strcpy_s. But not all calls to strcpy are safe, of course. Consider this code:

void GetIntegratedCutlery(char *out)
{
      strcpy(out, “spork”);
}

In Visual Studio 2005, even with template overloads enabled, this will give you a deprecation warning telling you it isn’t safe. strcpy isn’t told how big out is going to be.

Let’s imagine an improved version of the function          

void GetIntegratedCutleryEx(char *out, size_t size)
{
      strcpy_s(out, size, “spork”);
}

This function is now at least safe – strcpy_s has required us to tell strcpy how large the buffer out is.

A criticism I’ve heard of this kind of fix is that we have just ‘moved’ the problem, since now we have to ensure that the value passed to size is correct. This criticism has some validity – nothing about this code ensures that size has a correct value. One can argue that developers, especially maintenance ones, are more likely to get size correct when the have to write it explicitly. But this isn’t the strongest argument for this change. The best argument is that the buffer size is now explicit in the code and can be reasoned about by the toolset (as well as by developers themselves).

You may have noticed that the standard library headers have gotten much bigger in Visual C++ 2005. Compare the declaration of strstr from VC++ 2003:

_CRTIMP char *  __cdecl strstr(const char *, const char *);

with the one from VC++ 2005:

_CRTIMP __checkReturn _CONST_RETURN char *  __cdecl strstr(__in_z const char * _Str, __in_z const char * _SubStr);

There are three major changes here:

• _CONST_RETURN – this was added to bring us closer into conformance with the C++ standard.
• Parameter names (_Str, _SubStr) – these were added to improve the intellisense user experience in the IDE, as well as for readability
• Code annotations (__checkReturn, __in_z) – these annotations allow analysis tools to understand the intent of code better, and detect more issues

Annotations are the important ones for our purposes. __in_z actually tells the compiler quite a lot about the parameters of a function [see sal.h for the full definition], but at a high level it says “this is a input string (__in) that is null terminated (_z)”.

Annotations are used to find problems when you throw the /analyze switch on the compiler. For example, consider this function:

#include <stdlib.h>
#include <string.h>
#include <wchar.h>

wchar_t *AllocateAndFillW(size_t n, wchar_t c) throw(...)
{
      wchar_t *retVal=(wchar_t *)malloc(n*sizeof(wchar_t));   

      wmemset(retVal, c, n*sizeof(wchar_t)); /* line 9 */

      return retVal;
}

You can probably spot the bug on line 9 easily, but /analyze can do it for you:

f:\an.cpp(9) : warning C6383: buffer overrun due to conversion of an element count into a byte count: an element count is expected for parameter '3' in call to'wmemset'

[Note that /analyze is only supported in the Enterprise (team development)  versions of the product]

This happens because we’ve added annotations to malloc and wmemset in the CRT headers. /analyze uses those to see the problem. It sees that malloc returns __bcount_opt(_Size) a writable block of size bytes (bcount) which may be null (_opt). It sees that wmemset takes __out_ecount_full(_N) a writable buffer of _N elements. It then does the math and notices that the *sizeof(wchar_t) on line 9 is wrong and reports the problem. It found a buffer overrun for you.

Returning to our original example, there is no way the compiler can tell us whether this function has an overrun because it doesn’t know about the relationship between out and size

void GetIntegratedCutleryEx(char *out, size_t size)
{
      strcpy_s(out, size, “spork”);
}

However, we can teach it about this relationship:

void GetIntegratedCutleryEx(__out_ecount_z(size) char *out, __in size_t size)
{
      strcpy_s(out, size, “spork”);
}

And then when a user writes a bad piece of code:

void DeluxeExtendedCutleryEnumerationProviderManager(void)
{
      char b[3];

      GetIntegratedCutleryEx(b, _countof(b)+1);
}

You get a nice error

f:\an.cpp(23) : warning C6386: buffer overrun: accessing 'argument 1', the writable size is '3' bytes, but '4' bytes may be written: Lines: 21, 23

You can see how the deprecation warnings we add in the libraries (that encourage you to convert from strcpy to strcpy_s), and the annotations we added in the headers (which ensure you pass our functions the correct size) can be augmented by the annotations you write in your own code to further reduce the chance you’ll write errors (especially buffer overrun errors) in your code.

More next time. Looking forward to hearing from you.

Martyn

Last Friday I returned from the C standards committee meeting in Mont Tremblant, Canada at a beautiful hotel resort. These meetings are pretty fascinating, because of the diverse set of smart people they draw in. Even though C is quite stable at this point, there are always a range of interesting feature proposals and defect reports discussed.

I’ve been representing Visual C++ on this committee for a couple of years now, as part of my work on the Safe C and C++ Libraries [which you’ll hear more about]. My team started this work late in 2002, and I first took this to the committee back in April of 2003. There was very positive feedback, and over the last two years we’ve worked with the committee on a technical report which is now coming close to completion. The great thing about working with the committee on this project has been the opportunity to take some work we had done, get a lot of constructive and helpful feedback from a group of very knowledgeable people, and build up a standard which will help the whole C and C++ community migrate their existing code to work more safely. It’s been very fun, and a great learning experience.

You can find our implementation of these C functions, along with many others (C++, MFC, ATL), plus other safely improvements in Visual C++ 2005. We’ve tried to think what we could do to make your program safer, and that’s led to a bunch of features beyond the actual functions in question. I’ll talk about a few of these over the next few days.

One of my favourite features is our ability to automagically transform some less safe calls into more safe ones. For example, imagine that you have this code

void foo(char *str)
{

            char buf[20];

            strcpy(buf, str);

}

At this point, I hope that everyone reading already knows that this is an unsafe function. Function foo makes no attempt to ensure that str doesn’t overrun its buffer. VC8 is smart, and knows that strcpy isn’t safe, so it issues a warning when you compile this code, telling you that strcpy is unsafe and that you should use our new strcpy_s function instead. And you could do that. A basic fix for this function would be:

void foo(char *str)
{

            char buf[20];

            strcpy_s(buf, _countof(buf), str);

}

This fix isn’t very tolerant (since the new code still assumes that buf is large enough – the program’s invalid_parameter_handler will abort inside strcpy_s if not). A deep question when making this kind of code safer is whether you need a string to be truncated or to be  But it’s completely safe – this code will no longer buffer overrun.

But if you’re compiling C++, you can do something much simpler. You can simply put /D _CRT_SECURE_CPP_OVERLOAD_STANDARD_NAMES=1 on the compiler command line, and the original code will compile without a warning. We’ve used The Magic of Templates to transform the code into the safe code automatically.

These templates are not over-zealous. They only do what they know to be safe. For example, if you give us a function like this:

void bar(char *dest, char *src)
{

            strcpy(dest, src);

}

you'll still get the same warning you got above before you added the /D switch. The templates notice that dest is a pointer (not an array) whose size is unknown. So they allow the normal compiler warning to happen, instead of interposing themselves.

Clearly, the safest way to fix code like this is to switch to the use of a decent string class (I own at least two of these – std::string and CStringT). But much code won’t be able to be rewritten this deeply any time soon. So features like this one are intended to help those of you with large, existing code-bases to move your code forward to be safer, in much the same way that we at Microsoft have had to do over the last 6 years. We’ve taken all that we’ve learnt from our security pushes and secure development practices, and applied it to our library changes. We hope you find them helpful.

Write to me and let me know.

Martyn

[Edit 10/14/2005 MartynL: fix link to C starndard committee doc].

Hello.

Welcome to my second generation, re-engineered, upgraded weblog. I'm really excited to finally have enough time to return to this medium. My old blog lasted about a month, before I got busy and backlogged. I did keep collecting blog topics in the intervening two years, so I now have enough to talk about to last me till next year. I’ll try to keep this one going longer. Fingers crossed.

As you can tell from the heading, my name is Martyn Lovell. I'm a development lead in the Visual C++ group. I currently own the VC++ Libraries team, which is responsible for the C Runtime Library (CRT), Standard C++ Library (SCL), Microsoft Foundation Classes (MFC), Active Template Library (ATL). My team also own a bunch of new library code to help work with managed code -- I'll take some time to talk about this here in the future. As well as my work in libraries, I'm one of the leaders in Visual C++’s effort to allow seamless intermingling of managed and native code. Of course, like everyone else, I work on a few other problems too...

I've been at Microsoft for almost 10 years, all of it spent in the developer division. I started out working on the Developer Studio IDE, and then was one of the people who worked to architect a next generation IDE that could be an extensible platform for developer tools -- which became our current IDE. Next I was on Visual Studio Analyzer, a tool for understanding and visualising COM-based applications. For several years, I also owned the source control integration code in several IDEs, and was part of a project that was a precursor of the Visual Studio Team System.

The best thing about the development community now (compared to 10 years ago when I started at Microsoft) is the number of different ways we get to meet, interact with, listen to and understand our customers and partners. I’ve always enjoyed speaking at conferences and visiting developers for just this reason. But now we get to do so much more of this. Please keep sending me your questions, comments and feedback. Customer input had a major impact on the choices we made for Visual Studio 2005, and we’re now just starting the process of defining our next versions. So this is a great time to get in touch if you have ideas on what should come next for Visual C++.

Of course, not all of my time is spent at work. Like most blogs on MSDN, I'll mostly focus on technical issues here. You can find out more (too much) about me on my personal web site. Some of it is out of date, in keeping with the best traditions of personal web sites. You can find everything there -- pictures, links, reviews of concerts I've gone to, abstruse puzzles and an insanely long list of movies.

One of the best things about blogging is the two-way communication it encourages. Please do feel free to add comments, and to write to me (martynl@microsoft.com) if you have questions or comments on anything you find here, anything I work on or have worked on, or anything else that takes your fancy. I'm not always rapid at replying to email, but I do try to get there eventually. Of course, if you're looking for immediate help with a technical problem, you'll normally be best to start with our newsgroups. We have a very smart and highly motivated set of MVPs who answer questions very quickly. And if you need to report a bug in Visual Studio, then the Product Feedback Centre is the ideal place to do that. Bugs reported there rapidly get to the attention of the team involved, and we have great tracking mechanisms for these issues.

A reminder - everything here is my personal opinion, not that of Microsoft or of anyone else. These postings are provided "AS IS" with no warranties, and confer no rights. Consider yourself disclaimed.

Enough about me. On to the real content. I’m looking forward to hearing from you.

Martyn