Man vs Code

The Astoria team is hiring!

We have positions open for Developers and Program Managers.  Currently, we are finishing up V1 and planning for V2, so it is a great time to join the team.  For V2, we are thinking about expanding the project into new areas (e.g. offline support, sync, dynamic language support) so there will be plenty of new and exciting work.

Please forward your resume to aconrad@microsoft.com if you’re interested.  The project covers a diverse set of technical areas (databases, Linq, O/R, REST, etc) so we value passion for the technology more than deep expertise in any particular area. The Astoria team uses an agile development model, so the project moves fast and we encourage individuals to take on a variety of roles and responsibilities. If you’re a developer who wants to be responsible for more than just coding (i.e. API design, specification writing, working with customers and partners) this team is the place for you!

And finally, team members must be willing to participate in the grueling Team Astoria off-site events.

Update 4/23/08 - As of the start of this week, we now have a block of additional new positions on the Astoria Team we need to fill.  These include development, program management, and quality assurance positions.  We are trying to fill these very quickly, so if you are interested please contact us soon.

The news is out! At Mix, Windows Live will be announcing several new Live Services with AtomPub endpoints.  These services will also be compatible with Astoria and therefor will work with the Astoria client library.  An excerpt from David Treadwell's post:

Microsoft is making a large investment in unifying our developer platform protocols for services on the open, standards-based Atom format (RFC 4287) and the Atom Publishing Protocol (RFC 5023). At MIX we are enabling several new Live services with AtomPub endpoints which enable any HTTP-aware application to easily consume Atom feeds of photos and for unstructured application storage (see below for more details). Or you can use any Atom-aware public tools or libraries, such as .NET WCF Syndication to read or write these cloud service-based feeds.

In addition, these same protocols and the same services are now ADO.NET Data Services (formerly known as “ Project Astoria”) compatible. This means we now support LINQ queries from .NET code directly against our service endpoints, leveraging a large amount of existing knowledge and tooling shared with on-premise SQL deployments.

The intent for these early, experimental releases are to gather valuable feedback from the community around our idiomatic and freely licensed extensions to AtomPub which deal with important service scenarios, such as URL formats, nested directories, image streams, and service metadata. You can read more about this on the Project Astoria team blog.

There will also be a few more surprises in this area announced at MIX. Stay tuned. Try them out and give us your feedback!

At Mix, we have a Project Astoria talk all about using the Astoria client library to consume Live services. 

RESTful Data Services with the ADO.NET Data Services Framework

Wednesday, March 5 4:30 PM - 5:45 PM, Lando 4204

Speaker(s): Pablo Castro

Audience(s): Technical

Session Type: Breakout

Learn how to use ADO.NET Data Services Framework to easily create and consume REST data services on the web. This session will cover the main concepts of the ADO.NET Data Services Framework (aka Project "Astoria"), show how to use it, and discuss how to use it with Microsoft's broader vision of a common interface for Windows Live and 3rd party services.

So if you are going to Mix, be sure to go to that talk.  The Mix talks are also available online 24-48 hours after they are given.

On a related note Pablo, Mike, and myself will all be at Mix next week.  We should be spending a lot of time in the hands on lab/ Open Spaces area.  If anyone has any questions about any of our projects, wants to see some Astoria demos, etc - drop  me a line and we can set something up.

Last Wednesday we had a very important offsite for the entire Project Astoria team:

Many key technical issues were discussed. In fact, the entire team was quite spent after a long, hard day. 

Michael Sync has written a sample using the new Astoria Silverlight client library.

David Hayden posted a nice walk through for getting an Data Service up and running over Linq To Sql.

Our own Macelo Ruiz talks about some tweaks we made to the Astoria URI syntax on his personal blog.

Matthew Wills ported my original C# CopyToDataTable<T> Linq operator to VB.  According to him, he used Instant VB to convert the code from C# to VB and then hand tweaked it.  Looks like a fairly cool tool that we could use to port our C# unit tests to VB.   (We try to run the same functionals in both VB and C# these days because of the differences in Linq and comprehension support between the two compilers).

The updated Astoria Silverlight client is now available.

This client is 95% compatible with the client library we released back in the December CTP (we hope to make this 100% source code compatible by RTM) + works with the latest Silverlight 1.1 Alpha.

When the next preview of Silverlight releases this spring, we will release an update of the Astoria Silverlight client.

A number of people have asked me for a VB version of the CopyToDataTable<T> sample I wrote a few months back.   Unfortunately, between getting some skiing in (lots of snow in Washington this year!) and getting Astoria ready for Mix, I have not had very little free time.  Hence, if anyone else wants to take a crack at this - or knows of an existing port of the code to VB, let me know.

Marcelo Lopez Ruiz, a developer on the Astoria team, has written a nice post describing how to deal with Astoria server errors returned via HTTP.

One thing I regret that we didn't get into the Astoria December CTP was the ability to show detailed Astoria server errors to remote clients.  Something like what ASP.NET supports.  That way the developer can see what errors occurred on the server via the web browser.  Currently, one has to have a debugger attached to the server process to get the equivalent information.  This is particularity interesting in cases where the Astoria server code is not the component throwing the exception - for example when the Queryable provider throws an exception.

In addition, we are also looking at adding a way for providers to map specific exceptions to HTTP errors which could be used by provider writers to specify runtime HTTP errors instead of the generic "Internal Server Error".  For example, if there was a database constraint error when inserting some data.  Hopefully, this feature will make it into the next beta.

Jonathan Carter is in the midst of writing a 11 Part (!) Training Series about ADO.NET Data Services.

Guy Burnstein has also started a “Get Started with ADO.Net Data Services” post series.

Late Sunday night we released the first public preview of ASP.NET 3.5 Extensions.  For information on the release, see ScottGu's announcement.

As part of that preview, we released the first CTP of the production version of Project Astoria.  For details, see public release announcement by Mike Flasko on the Astoria Team Blog.

Astoria actually has been given its official Microsoft name:  ADO.NET Data Services.  So, in this post I will use the name Astoria and ADO.net Data Services interchangeably.

For a complete overview of Astoria check out the information available on Microsoft Live Labs.

For the next few weeks, I will be writing a series of blog posts to demonstrate what I think is one of the most compelling features in Astoria, namely remote execution of Linq queries via the RESTFUL API exposed by Astoria Data Services.

For these posts, I will be covering:

1)  How to set up an Astoria Data Service which can be used for remote execution of Linq queries.

2)  Dive deep into the Astoria Client API's Linq support including a detailed description of how Linq queries are translated to URIs.

3)  How to make the Astoria Data Service updateable so the results of the Linq queries can be remotely modified and persisted.

So the goal of this initial post is to set up an Astoria Data Service over a Linq to Sql DataContext source and remotely execute a Linq query via the Astoria client library over HTTP.

(The following steps assume Visual Studio 2008 and ASP.NET 3.5 Extensions has been installed.  Also, this demo assumes access to the Sql Server sample Northwind database).

First, Create a standard ASP.NET Web Application:

Next, add an ADO.NET Data Service called Northwind.svc to the project via the Add New Item option:

Next, add a new set of Linq to Sql via the designer called Northwind.dbml.

From my Data Connections in the Server Explorer, I add the Categories, Products, and Supplier tables to Northwind.dbml from the Northwind database

Because the keys for the new types follow the Astoria convention of being named [EntityName]Id, I don't have to do anything further.  The Linq to Sql data classes and the strongly typed DataContext (NorthwindDataContext) is totally useable with Astoria without any modifications.

Next, I want to wire up my Astoria Data Service to use NorthwindDataContext as the data source.  To do this, go back to the Northwind.svc.cs file that was created when the Astoria Data Service was added to the project.  Look for the TODO's in the Northwind class:

    public class Northwind : WebDataService< /* TODO: put your data source class name here */ >
    {
        // This method is called once during service initialization to allow
        // service-specific policies to be set
        public static void InitializeService(IWebDataServiceConfiguration config)
        {
            // TODO: set rules to indicate which entity sets and service operations are
            // visible, updatable, etc.
            // (for testing purposes use "*" to indicate all entity sets/service
            // operations, but that option should NOT be used in production systems)

            // Example for entity sets (this example uses "AllRead" which allows reads but not writes)
            // config.SetResourceContainerAccessRule("MyEntityset", ResourceContainerRights.AllRead);

            // Example for service operations
            // config.SetServiceOperationAccessRule("MyServiceOperation", ServiceOperationRights.All);
        }

        // Query interceptors, change interceptors and service operations go here
    }

First, put the name of the DataContext in as the generic parameter for the Northwind class.

Next. add the following line to the InitializeService method:

config.SetResourceContainerAccessRule("*", ResourceContainerRights.AllRead);

By default, Astoria turns off all access to the data source.  By adding this line, I am giving read access to all public IQueryable<T> entry points on the DataContext. 

Note, the wildcard option (*) should only be used for testing purposes.  For actual application development, please explicitly grant access to each IQueryable<T> entry point by via the property name.

Now build the application and press F5 to fire it up.  If everything went ok, you should get back the following Atom service definition when you access the URI http://localhost:18752/Northwind.svc/ (I am using port 18752 with Cassini for debugging).

  <service xml:base="http://localhost:18752/Northwind.svc/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:app="http://www.w3.org/2007/app" xmlns="http://www.w3.org/2007/app">
    <workspace>
       <atom:title>Default</atom:title> 
       <collection href="Categories">
         <atom:title>Categories</atom:title> 
       </collection>
       <collection href="Products">
         <atom:title>Products</atom:title> 
       </collection>
       <collection href="Suppliers">
         <atom:title>Suppliers</atom:title> 
       </collection>
    </workspace>
  </service>

Now, try a few URIs to make sure everything it working:

(Note, Internet Explorer by default renders ATOM in a friendly format that makes Astoria feeds unreadable.  To fix this, turn off IE feed viewing by turning off the option :  Tools -> Internet Options -> Content ->  Feeds (settings) -> Turn on feed viewing)

http://localhost:18752/Northwind.svc/Products - gets all the Products

http://localhost:18752/Northwind.svc/Products(2) - gets the Product with ProductId == 2

http://localhost:18752/Northwind.svc/Products?$filter=UnitsInStock gt 100 - gets all the Products where UnitsInStock is greater than 100

http://localhost:18752/Northwind.svc/Products?$orderby=ProductName&$top=10 - orders the Products by ProductName and gets the first 10.

For more information about Astoria's URI Addressing Scheme see here.

Now, lets get a sample Linq query working.

First, add another project to the solution.  In this case, a simple console Application will work:

In this project, add a reference to the Astoria client library assembly called Microsoft.Data.WebClient.dll. It should be in your /Program Files/Reference Assemblies/Microsoft/Framework/ASP.NET 3.5 Extensions directory.

Now add make sure the namespace Microsoft.Data.WebClient is in scope, and add the following class to your project:

    [OpenObject("PropBag")]
    public class Product
    {
        private Dictionary<string, object> propBag = new Dictionary<string, object>();

        [Key]
        public int ProductID { get; set; }
        
        public string ProductName { get; set; }
        
        public int UnitsInStock { get; set; }

        public IDictionary<string, object> PropBag { get { return propBag; } }

    }

Finally, add the following to Program.Main:

        static void Main(string[] args)
        {
            WebDataContext context = new WebDataContext("http://localhost:18752/Northwind.svc");

            var query = from p in context.CreateQuery<Product>("Products")
                        where p.UnitsInStock > 100
                        select p;

            foreach (Product p in query)
            {
                Console.WriteLine(p.ProductName + " , UnitsInStock= " + p.UnitsInStock);
            }

        }

To get both projects to start up for debugging, select Solution-> Properties and select Multiple startup projects with the Action of Start.

Put a breakpoint on the foreach statement in the client app, and hit F5.

If you hover over the query variable, you will actually see the Astoria URI which the Linq query is translated into by the Astoria client library:

http://localhost:18752/Northwind.svc/Products?$filter=(UnitsInStock)%20gt%20(100)

So, there you go.  Linq to Astoria's RESTFUL API.  In other words, Linq to REST. 

In future posts, I will explain how that worked.  What all the Linq capabilities that are supported.  And what one can do with the objects once they are materialized.

A couple of months ago I became Development Lead for Project Astoria.   The goal of Project Astoria is to provide a data services for the web. AKA Project Astoria provides a RESTFUL head for any sort of "queryable" data, for example data stored in relational databases.  It also contains a O/R style client library (usable via a standard .Net or Silverlight client) with Linq support for accessing the Astoria Data Services.  For a much more in depth description of the project, check out the project overview document.

Over the last few months, the team has moved from prototype mode to working on the production code.  In an effort to elicit quicker feedback on our design, we have started publishing our design notes on the Astoria Team Blog.  Normally our design team meets twice a week for a couple of hours, so we expect to be able to publish these notes fairly regularly.  In most cases, we are going to be right in the middle of coding and testing the features described in the design notes when the notes are published.   Hence, based on feedback from the developer community we will change the design and implementation mid flight. 

Pablo has already published the first set of design notes on the Payload Formats.  Later this month we should have notes on the URI format and the service extensibility.

We are also looking to hire a few people in anyone is interested.  See here for more information or send me your resume directly to me if interested.

Tok Wee Hyong of the .Net Singapore user group has put together a nice Project Jasper sample available here if anyone is interested.

In the original Linq CTP and the first Orcas Beta, we included a DataSet specific Linq operator called CopyToDataTable<T> (It was called ToDataTable at one point also).  For Beta 2 of Orcas, we ended up restricting this method to only work with DataRows (or some derived type) via a generic constraint on the method. 

The reason for this was simply resource constraints.  When we started to design how the real version of CopyToDataTable<T> should work, we realized that there are a number of potentially interesting mappings between objects and DataRows and didn't have the resources to come up with a complete solution.  Hence, we decided to cut the feature and release the source as a sample.

Surprising to us, a lot of folks noticed this and were wondering where the feature had gone.  It does make a nice solution for dealing with projections in Linq in that one can load instances of anonymous types into DataRows. 

So as promised, below is sample code of how to implement CopyToDataTable<T> when the generic type T is not a DataRow. 

A few notes about this code:

1.  The initial schema of the DataTable is based on schema of the type T.  All public property and fields are turned into DataColumns.

2.  If the source sequence contains a sub-type of T, the table is automatically expanded for any addition public properties or fields.

3.  If you want to provide a existing table, that is fine as long as the schema is consistent with the schema of the type T.

4.  Obviously this sample probably needs some perf work.  Feel free to suggest improvements.

5.  I only included two overloads - there is no technical reason for this, just Friday afternoon laziness.

    class Sample
    {
        static void Main(string[] args)
        {
            // create sequence 
            Item[] items = new Item[] { new Book{Id = 1, Price = 13.50, Genre = "Comedy", Author = "Jim Bob"}, 
                                        new Book{Id = 2, Price = 8.50, Genre = "Drama", Author = "John Fox"},  
                                        new Movie{Id = 1, Price = 22.99, Genre = "Comedy", Director = "Phil Funk"},
                                        new Movie{Id = 1, Price = 13.40, Genre = "Action", Director = "Eddie Jones"}};

                        
            var query1 = from i in items
                         where i.Price > 9.99
                         orderby i.Price
                         select i;

            // load into new DataTable
            DataTable table1 = query1.CopyToDataTable();

            // load into existing DataTable - schemas match            
            DataTable table2 = new DataTable();
            table2.Columns.Add("Price", typeof(int));
            table2.Columns.Add("Genre", typeof(string));

            var query2 = from i in items
                         where i.Price > 9.99
                         orderby i.Price
                         select new {i.Price, i.Genre};

            query2.CopyToDataTable(table2, LoadOption.PreserveChanges);


            // load into existing DataTable - expand schema + autogenerate new Id.
            DataTable table3 = new DataTable();
            DataColumn dc = table3.Columns.Add("NewId", typeof(int));
            dc.AutoIncrement = true;
            table3.Columns.Add("ExtraColumn", typeof(string));

            var query3 = from i in items
                         where i.Price > 9.99
                         orderby i.Price
                         select new { i.Price, i.Genre };

            query3.CopyToDataTable(table3, LoadOption.PreserveChanges);

            // load sequence of scalars.

            var query4 = from i in items
                         where i.Price > 9.99
                         orderby i.Price
                         select i.Price;

            var DataTable4 = query4.CopyToDataTable();
        }

        public class Item
        {
            public int Id { get; set; }
            public double Price { get; set; }
            public string Genre { get; set; }   
        }

        public class Book : Item
        {
            public string Author { get; set; }
        }

        public class Movie : Item
        {
            public string Director { get; set; }
        }
        
    }

    public static class DataSetLinqOperators
    {
        public static DataTable CopyToDataTable<T>(this IEnumerable<T> source)
        {
            return new ObjectShredder<T>().Shred(source, null, null);
        }

        public static DataTable CopyToDataTable<T>(this IEnumerable<T> source, 
                                                    DataTable table, LoadOption? options)
        {
            return new ObjectShredder<T>().Shred(source, table, options);
        }

    }

    public class ObjectShredder<T>
    {
        private FieldInfo[] _fi;
        private PropertyInfo[] _pi;
        private Dictionary<string, int> _ordinalMap;
        private Type _type;

        public ObjectShredder()
        {
            _type = typeof(T);
            _fi = _type.GetFields();
            _pi = _type.GetProperties();
            _ordinalMap = new Dictionary<string, int>();
        }

        public DataTable Shred(IEnumerable<T> source, DataTable table, LoadOption? options)
        {
            if (typeof(T).IsPrimitive)
            {
                return ShredPrimitive(source, table, options);   
            }
    

            if (table == null)
            {
                table = new DataTable(typeof(T).Name);
            }

            // now see if need to extend datatable base on the type T + build ordinal map
            table = ExtendTable(table, typeof(T));

            table.BeginLoadData();
            using (IEnumerator<T> e = source.GetEnumerator())
            {
                while (e.MoveNext())
                {
                    if (options != null)
                    {
                        table.LoadDataRow(ShredObject(table, e.Current), (LoadOption)options);
                    }
                    else
                    {
                        table.LoadDataRow(ShredObject(table, e.Current), true);
                    }
                }
            }
            table.EndLoadData();
            return table;
        }

        public DataTable ShredPrimitive(IEnumerable<T> source, DataTable table, LoadOption? options)
        {
            if (table == null)
            {
                table = new DataTable(typeof(T).Name);
            }

            if (!table.Columns.Contains("Value"))
            {
                table.Columns.Add("Value", typeof(T));
            }

            table.BeginLoadData();
            using (IEnumerator<T> e = source.GetEnumerator())
            {
                Object[] values = new object[table.Columns.Count];
                while (e.MoveNext())
                {
                    values[table.Columns["Value"].Ordinal] = e.Current;

                    if (options != null)
                    {
                        table.LoadDataRow(values, (LoadOption)options);
                    }
                    else
                    {
                        table.LoadDataRow(values, true);
                    }
                }
            }
            table.EndLoadData();  
            return table; 
        }

        public DataTable ExtendTable(DataTable table, Type type)
        {
            // value is type derived from T, may need to extend table.
            foreach (FieldInfo f in type.GetFields())
            {
                if (!_ordinalMap.ContainsKey(f.Name))
                {
                    DataColumn dc = table.Columns.Contains(f.Name) ? table.Columns[f.Name]
                        : table.Columns.Add(f.Name, f.FieldType);
                    _ordinalMap.Add(f.Name, dc.Ordinal);               
                }
            }
            foreach (PropertyInfo p in type.GetProperties())
            {
                if (!_ordinalMap.ContainsKey(p.Name))
                {
                    DataColumn dc = table.Columns.Contains(p.Name) ? table.Columns[p.Name]
                        : table.Columns.Add(p.Name, p.PropertyType);
                    _ordinalMap.Add(p.Name, dc.Ordinal);
                }
            }
            return table;
        }

        public object[] ShredObject(DataTable table, T instance)
        {

            FieldInfo[] fi = _fi;
            PropertyInfo[] pi = _pi;

            if (instance.GetType() != typeof(T))
            {
                ExtendTable(table, instance.GetType());
                fi = instance.GetType().GetFields();
                pi = instance.GetType().GetProperties();
            }

            Object[] values = new object[table.Columns.Count];
            foreach (FieldInfo f in fi)
            {
                values[_ordinalMap[f.Name]] = f.GetValue(instance);
            }

            foreach (PropertyInfo p in pi)
            {
                values[_ordinalMap[p.Name]] = p.GetValue(instance, null);
            }
            return values;
        }
    }

Over the last few months, it dawned on us that if we were writing software apis/ frameworks (Jasper, Astoria) specifically designed for Agile development that perhaps we should really understand Agile development by actually doing it. Breath taking insight, I know…

So here is the problem. I need to learn about Agile development quickly and efficiently. (I need to be agile about learning about Agile development). I have started reading Robert Martin’s book, but am also interested in any pointers, feedback, or opinions folks might have. My first task is to figure out which flavor of Agile development we are going to do.

Scott Hanselman has blogged about what he thinks is the Six Essential Language Agnostic Programming Books.  Can't say I can argue with any of the six he has choices, but let me make a few suggestions:

Patterns of Enterprise Application Architecture – Martin Fowler

My own personal opinion, may now be more important than the original Design Patterns. 

Introduction to Functional Programming using Haskell

I actually have the original version of the book that uses a pseudo language instead of Haskell. Not entirely language agnostic since it is a book about functional programming, but required reading to understand in depth any of a number of the modern trends in the industry.

Algorithms in C++, Robert Sedgewick

Ok, the quintessential college text book still on most people's bookshelf.   And we all know that even though it isn't language agnostic, we all us it in a agnostic way.  Over the last twelve years, I have now used the book as a reference for developing in eight distinct languages (C, C++, Java, VB, C#, VB.NET, Ruby, Python)  Looks like Sedgewick also has versions of the book for Java now.

On a random, but related note - a little bit of Andrew Conrad trivia.  I did my graduate work at Seattle University in the same program attended by Steve McConnell (author of Code Complete, first book in Scott's list).  In fact, during the last year we did a year long project which was sponsored by Steve McConnell and his company (Construx) where they acted as the customer and we were the development team.  So I actually had a bunch of meetings with Steve where we argued app features and schedule.