Building a .NET Disassembler (Part 4) – Reading the Metadata Tables in the #~ Stream

We’re finally getting to something that isn’t just reading headers into a data structure.

Last time, in Part 3 of this series, we read in the “.text” section header, and the stream headers. One of those streams contains all the metadata, and is called, appropriately, the “Metadata Stream”. This stream has the name of either "#~" or "#-". There will be one or the other, never both. The difference is that #~ indicates that this will be a compressed/optimized metadata stream, and #- indicates an uncompressed/unoptimized metadata stream.

So lets get to work reading this metadata stream. First we use the metadata stream header where the name is “#~” or “#-” to get the bytes from the file that makes up this stream. Remember that the .Offset indicated by the stream header is an RVA (relative virtual address) relative to the metadataDirectoryAddress in the CLRHeader. So to get the byte[] for any of the streams from its stream header, we can do this:

        public byte[] GetStreamBytes(BinaryReader reader, StreamHeader streamHeader, uint metadataDirectoryAddress, IEnumerable<Section> sections)
        {
            var rva = metadataDirectoryAddress + streamHeader.Offset;
            var fileOffset = AddressingUtils.RelativeVirtualAddressToFileOffset(rva, sections);
            reader.BaseStream.Seek((long)fileOffset, SeekOrigin.Begin);
            return reader.ReadBytes((int)streamHeader.Size);
        }

The parameters are:

  • BinaryReader reader = The reader opened to read the entire assembly file.
  • StreamHeader streamHeader = The header of the stream we want to read.
  • uint metadataDirectoryAddress = The Metadata address from the TextSection.CLRHeader.
  • IEnumerable
    sections = The list of all the section headers. Even though the steam should exist within the .text section only, who knows what some crazy obfuscator might try.

So that will give us the entire byte[] of the stream we want to read. In my case, I am then passing the bytes for the metadata stream into a separate “reader” class:

    public class MetadataStreamReader
    {
        private readonly MetadataReader _reader;

        public MetadataStreamReader(byte[] data)
        {
            _reader = new MetadataReader(new MemoryStream(data));
        }

        public MetadataStream Read()
        {
            // code to parse the metadata will go here...
        }

So you can see I am reading the bytes from the file, bringing them in to memory, and making a new BinaryReader around just the stream. This obviously isn’t the most efficient thing ever, but… Maybe I will refactor it later. You could pass in the original reader that has the entire file open, just make sure you did a .Seek() to the beginning of the #~ stream first.

The contents of the metadata stream is laid out like this:

Metadata Stream Layout

Reading the Metadata Header

Yep, the first set of bytes in the #~ stream is another header. It looks like this:

    /// <summary>
    /// Header for the #~ stream.
    /// </summary>
    public class MetadataStreamHeader
    {
        public uint Reserved1 { get; set; }
        public byte MajorVersion { get; set; }
        public byte MinorVersion { get; set; }
        public StreamOffsetSizeFlags OffsetSizeFlags { get; set; } // indicates offset sizes to be used within the other streams.
        public byte Reserved2 { get; set; } // Always set to 0x01 [01]
        public MetadataTableFlags TablesFlags { get; set; } // indicated which tables are present. 8 bytes.
        public MetadataTableFlags SortedTablesFlags { get; set; } // indicated which tables are sorted. 8 bytes.
        public uint[] TableSizes { get; set; } // Size of each table. Array count will be same as # of '1's in TableFlags.
    }

Note that the last property here is actually the 2nd thing in my diagram image above, the row counts for each metadata table. Just bear with me, and I’ll get to the details…

Anyway, Reading the first couple fields is easy…

        private MetadataStreamHeader ReadHeader()
        {
            var header = new MetadataStreamHeader
            {
                Reserved1 = _reader.ReadUInt32(),
                MajorVersion = _reader.ReadByte(),
                MinorVersion = _reader.ReadByte(),
                OffsetSizeFlags = (StreamOffsetSizeFlags)_reader.ReadByte(),
                Reserved2 = _reader.ReadByte(),
                TablesFlags = (MetadataTableFlags)_reader.ReadUInt64(),
                SortedTablesFlags = (MetadataTableFlags)_reader.ReadUInt64(),
            };

You will notice some enums being used here, so let me explain those…

Determining #Strings, #US, and #Blob Offset Sizes

The first enum you run into is with the “OffsetSizeFlags” property. It is a single byte, but only 3 bits here actually matter. As we get into the metadata tables, they contain references into the other streams (#Strings, #GUID, #Blob). Each of these offsets could normally be 4 bytes (uint), but how often are there really THAT many strings, guids, etc? So to save space, the crafty people at Microsoft decided to use 3 bits in this flag to indicate what size the offsets are, 2-byte or 4-byte. So we can make a StreamOffsetSizeFlags enum that looks like this:

    /// <summary>
    /// If the flag is not set, the offsets into the respective heap are stored as 2-bytes,
    /// If the flag is set, then the offsets are stored as 4-bytes.
    /// </summary>
    [Flags]
    public enum StreamOffsetSizeFlags : byte
    {
        String = 0x01,
        GUID = 0x02,
        Blob = 0x04,
    }

What gets annoying about this is that now for all the rest of the metadata stream, you need to keep this in mind when reading the data (do I read 2 bytes or 4 bytes?).

To make this easier on myself, I made a new class:

    public class MetadataReader : BinaryReader
    {
        public StreamOffsetSizeFlags StreamOffsetSizeFlags { get; set; }

        public MetadataReader(Stream input) : base(input)
        {
        }

        public MetadataReader(Stream input, Encoding encoding)
            : base(input, encoding)
        {
        }

        public uint ReadStringStreamIndex()
        {
            return ReadStreamIndex(StreamOffsetSizeFlags.String);
        }

        public uint ReadGuidStreamIndex()
        {
            return ReadStreamIndex(StreamOffsetSizeFlags.GUID);
        }

        public uint ReadBlobStreamIndex()
        {
            return ReadStreamIndex(StreamOffsetSizeFlags.Blob);
        }

        private uint ReadStreamIndex(StreamOffsetSizeFlags streamFlag)
        {
            return StreamOffsetSizeFlags.HasFlag(streamFlag) ? ReadUInt32() : ReadUInt16();
        }
    }

I don’t entirely like it right now because it introduces some temporal coupling in another spot, but it is OK for explanatory purposes…

The Metadata is Composed of Several Tables.

The rest of the metadata stream is made up of metadata tables. The ECMA-335 spec defines these tables, however if a table is empty, then it is not included in the metadata stream.

When we read the metadata stream header, we saw an 8-byte propery named “TablesFlags”. So this gives us 64 bitt flags. Each of which tells us if a particular table, represented by that bit, exists in the metadata stream. This means that .NET can have at most 64 metadata tables. Currently, the ECMA-335 spec defines:

    [Flags]
    public enum MetadataTableFlags : ulong
    {
        Module = 1,
        TypeRef = 2,
        TypeDef = 4,
        Reserved1 = 8,
        Field = 16,
        Reserved2 = 32,
        Method = 64,
        Reserved3 = 128,
        Param = 256,
        InterfaceImpl = 512,
        MemberRef = 1024,
        Constant = 2048,
        CustomAttribute = 4096,
        FieldMarshal = 8192,
        DeclSecurity = 16384,
        ClassLayout = 32768,
        FieldLayout = 65536,
        StandAloneSig = 131072,
        EventMap = 262144,
        Reserved4 = 524288,
        Event = 1048576,
        PropertyMap = 2097152,
        Reserved5 = 4194304,
        Property = 8388608,
        MethodSemantics = 16777216,
        MethodImpl = 33554432,
        ModuleRef = 67108864,
        TypeSpec = 134217728,
        ImplMap = 268435456,
        FieldRVA = 536870912,
        Reserved6 = 1073741824,
        Reserved7 = 2147483648,
        Assembly = 4294967296,
        AssemblyProcessor = 8589934592,
        AssemblyOS = 17179869184,
        AssemblyRef = 34359738368,
        AssemblyRefProcessor = 68719476736,
        AssemblyRefOS = 137438953472,
        File = 274877906944,
        ExportedType = 549755813888,
        ManifestResource = 1099511627776,
        NestedClass = 2199023255552,
        GenericParam = 4398046511104,
        MethodSpec = 8796093022208,
        GenericParamConstraint = 17592186044416,
    }

You will notice I included some “ReservedX” table names that are undefined, but represented a gap in the sequence, so I left them in there as placeholders.

So ECMA currently (as of .NET 4 I think) defines 38 (of the possible 64) tables, and we use the masks above to see which are included in this metadata stream.

The metadata header also contained a “SortedTablesFlags” property with the same set of flags, which indicates which of these tables, if any, are sorted. I’m not entirely sure what that means yet…

Determine Metadata Table Sizes (Row Counts)

Immediately following the metadata header, is an array of 4-byte uints, each of which specifies the row count for a metadata table. The order and number of these uints is exactly the same as the order (low to high bit) of the metadata table existence flags.

So for example, if MetadataStreamHeader.TablesFlags = 0x13 then that means it has the binary value “10011” which is equivalent to the MetadataTableFlags Module | TypeRef | Field. So first we want to know how many metadata tables there will be. We can see fromt eh binary representation that there are 3 tables (3 bits set to 1). However in C#, here is a little helper:

        public int GetTableCount(MetadataTableFlags tablesFlags)
        {
            var count = 0;
            while (tablesFlags != 0)
            {
                tablesFlags = tablesFlags & (tablesFlags - 1);
                count++;
            }
            return count;
        }

So that will return 3 for our example flags, so we know we have 3 metadata tables to read. That means that there will be 3 4-byte uints right after the header, representing the row counts for the respective tables, in the order Module, then TypeRef, then Field.

In C# it looks something like this:

            var numberOfTables = GetTableCount(header.TablesFlags);
            header.TableSizes = new uint[numberOfTables];
            for (var i = 0; i < numberOfTables; i++)
            {
                header.TableSizes[i] = _reader.ReadUInt32();
            }

Great! So now we know how many tables we have, and how many rows are in each of those tables.

Reading the Individual Metadata Table Rows

Right after all the table row counts are the actual metadata rows. The first table included, “Module” in our example, is up first, so we read however many rows from that as were specified. Then the TypeRef table will immediately follow, with however many rows were specified. Then the Field table.

So for each metadata table row, you need to know which table you are reading. Every tabl;e has a different layout, defined in the ECMA-335 spec. You will need to refer to it to get the details.

For this example, I will just show the 1st table, “Module”.

Each row in the Module table looks like this:

    /// <summary>
    /// The rows in the Module table result from .module directives in the Assembly.
    /// Defined in ECMA-335 II.22.30
    /// </summary>
    public class ModuleTableRow : IMetadataTableRow
    {
        /// <summary>
        /// a 2-byte value, reserved, shall be zero
        /// </summary>
        public ushort Generation { get; set; }

        /// <summary>
        /// an index into the String heap
        /// </summary>
        public uint Name { get; set; }

        /// <summary>
        /// an index into the Guid heap; simply a Guid used to distinguish between two versions of the same module
        /// </summary>
        public uint Mvid { get; set; }

        /// <summary>
        /// an index into the Guid heap; reserved, shall be zero
        /// </summary>
        public uint EncId { get; set; }

        /// <summary>
        /// an index into the Guid heap; reserved, shall be zero
        /// </summary>
        public uint EncBaseId { get; set; }

        public void Read(MetadataReader reader)
        {
            Generation = reader.ReadUInt16();
            Name = reader.ReadStringStreamIndex();
            Mvid = reader.ReadGuidStreamIndex();
            EncId = reader.ReadGuidStreamIndex();
            EncBaseId = reader.ReadGuidStreamIndex();
        }
    }

You can see here that I am defining the indexes into the other streams as uints, even though they might only be ushorts in the actual file, but my custom MetadataReader is handling that for me.

So that is basically it… Refer to the ECMA spec for each of the 38 defined tables, and make a class and a method to read it in. Then read however many rows were specified by the table sizes array.

Next Time…

Next time we are going to read the #Strings stream. It is pretty easy, but I wanted to make it a separate part of this series.

See you next time!

Advertisements
Tagged with: , ,
Posted in Programming
One comment on “Building a .NET Disassembler (Part 4) – Reading the Metadata Tables in the #~ Stream
  1. […] To get the byte[] that makes up this stream, see the previous part on reading the metadata stream. […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

CodingWithSpike is Jeff Valore. A professional software engineer, focused on JavaScript, Web Development, C# and the Microsoft stack. Jeff is currently a Software Engineer at Virtual Hold Technologies.


I am also a Pluralsight author. Check out my courses!

%d bloggers like this: