BSON Tutorial

Writing

To create a ByteBuffer for writing (i.e. serializing to BSON), instantiate BSON::ByteBuffer with no arguments:

buffer = BSON::ByteBuffer.new

To write raw bytes to the byte buffer with no transformations, use put_byte and put_bytes methods. They take a byte string as the argument and copy this string into the buffer. put_byte enforces that the argument is a string of length 1; put_bytes accepts any length strings. The strings can contain null bytes.

buffer.put_byte("\x00")
buffer.put_bytes("\xff\xfe\x00\xfd")

Subsequent write methods write objects of particular types in the BSON spec. Note that the type indicated by the method name takes precedence over the type of the argument - for example, if a floating-point value is given to put_int32, it is coerced into an integer and the resulting integer is written to the byte buffer.

To write a UTF-8 string (BSON type 0x02) to the byte buffer, use put_string:

buffer.put_string("hello, world")

Note that BSON strings are always encoded in UTF-8. Therefore, the argument must be either in UTF-8 or in an encoding convertable to UTF-8 (i.e. not binary). If the argument is in an encoding other than UTF-8, the string is first converted to UTF-8 and the UTF-8 encoded version is written to the buffer. The string must be valid in its claimed encoding, including being valid UTF-8 if the encoding is UTF-8. The string may contain null bytes.

The BSON specification also defines a CString type, which is used for example for document keys. To write CStrings to the buffer, use put_cstring:

buffer.put_cstring("hello, world")

As with regular strings, CStrings in BSON must be UTF-8 encoded. If the argument is not in UTF-8, it is converted to UTF-8 and the resulting string is written to the buffer. Unlike put_string, the UTF-8 encoding of the argument given to put_cstring cannot have any null bytes, since the CString serialization format in BSON is null terminated.

Unlike put_string, put_cstring also accepts symbols and integers. In all cases the argument is stringified prior to being written:

buffer.put_cstring(:hello)
buffer.put_cstring(42)

To write a 32-bit or a 64-bit integer to the byte buffer, use put_int32 and put_int64 methods respectively. Note that Ruby integers can be arbitrarily large; if the value being written exceeds the range of a 32-bit or a 64-bit integer, put_int32 and put_int64 raise RangeError.

buffer.put_int32(12345)
buffer.put_int64(123456789012345)

To write a 64-bit floating point value to the byte buffer, use put_double:

buffer.put_double(3.14159)

To obtain the serialized data as a byte string (for example, to send the data over a socket), call to_s on the buffer:

buffer = BSON::ByteBuffer.new
buffer.put_string('testing')
socket.write(buffer.to_s)

Reading

To create a ByteBuffer for reading (i.e. deserializing from BSON), instantiate BSON::ByteBuffer with a byte string as the argument:

buffer = BSON::ByteBuffer.new(string) # a read mode buffer.

Reading from the buffer is done via the following API:

buffer.get_byte # Pulls a single byte from the buffer.
buffer.get_bytes(value) # Pulls n number of bytes from the buffer.
buffer.get_cstring # Pulls a null-terminated string from the buffer.
buffer.get_double # Pulls a 64-bit floating point from the buffer.
buffer.get_int32 # Pulls a 32-bit integer (4 bytes) from the buffer.
buffer.get_int64 # Pulls a 64-bit integer (8 bytes) from the buffer.
buffer.get_string # Pulls a UTF-8 string from the buffer.

To restart reading from the beginning of a buffer, use rewind:

buffer.rewind

BSON::Binary

Use BSON::Binary objects to store arbitrary binary data. The Binary objects can be constructed from binary strings as follows:

BSON::Binary.new("binary_string")
# => <BSON::Binary:0x47113101192900 type=generic data=0x62696e6172795f73...>

By default, Binary objects are created with BSON binary subtype 0 (:generic). The subtype can be explicitly specified to indicate that the bytes encode a particular type of data:

BSON::Binary.new("binary_string", :user)
# => <BSON::Binary:0x47113101225420 type=user data=0x62696e6172795f73...>

Valid subtypes are :generic, :function, :old, :uuid_old, :uuid, :md5 and :user.

The data and the subtype can be retrieved from Binary instances using data and type attributes, as follows:

binary = BSON::Binary.new("binary_string", :user)
binary.data
=> "binary_string"
binary.type
=> :user

str = "binary_string"
str.encoding
# => #<Encoding:US-ASCII>
binary = BSON::Binary.new(str)
binary.data
# => "binary_string"
binary.data.encoding
# => #<Encoding:ASCII-8BIT>

uuid_str = "00112233-4455-6677-8899-aabbccddeeff"
BSON::Binary.from_uuid(uuid_str)
# => <BSON::Binary:0x46986653612880 type=uuid data=0x0011223344556677...>

binary = BSON::Binary.new("\x00\x11\x22\x33\x44\x55\x66\x77\x88\x99\xAA\xBB\xCC\xDD\xEE\xFF".force_encoding('BINARY'), :uuid)
=> <BSON::Binary:0x46942046606480 type=uuid data=0x0011223344556677...>
binary.to_uuid
=> "00112233-4455-6677-8899aabbccddeeff"

binary = BSON::Binary.from_uuid(uuid_str, :standard)
binary.to_uuid(:standard)

binary = BSON::Binary.new("\x00\x11\x22\x33\x44\x55\x66\x77\x88\x99\xAA\xBB\xCC\xDD\xEE\xFF".force_encoding('BINARY'), :uuid_old)
=> <BSON::Binary:0x46942046606480 type=uuid data=0x0011223344556677...>
binary.to_uuid
# => ArgumentError (Representation must be specified for BSON::Binary objects of type :uuid_old)
binary.to_uuid(:csharp_legacy)
# => "33221100-5544-7766-8899aabbccddeeff"
binary.to_uuid(:java_legacy)
# => "77665544-3322-1100-ffeeddccbbaa9988"
binary.to_uuid(:python_legacy)
# => "00112233-4455-6677-8899aabbccddeeff"

uuid_str = "00112233-4455-6677-8899-aabbccddeeff"
BSON::Binary.from_uuid(uuid_str, :csharp_legacy)
# => <BSON::Binary:0x46986653650480 type=uuid_old data=0x3322110055447766...>
BSON::Binary.from_uuid(uuid_str, :java_legacy)
# => <BSON::Binary:0x46986653663960 type=uuid_old data=0x7766554433221100...>
BSON::Binary.from_uuid(uuid_str, :python_legacy)
# => <BSON::Binary:0x46986653686300 type=uuid_old data=0x0011223344556677...>

BSON::Binary.from_uuid('77665544-3322-1100-ffeeddccbbaa9988',:java_legacy).to_uuid(:csharp_legacy)
# => "33221100-5544-7766-8899aabbccddeeff"

BSON::Code

Represents a string of JavaScript code.

BSON::Code.new("this.value = 5;")

BSON::CodeWithScope

Represents a string of JavaScript code with a hash of values.

BSON::CodeWithScope.new("this.value = age;", age: 5)

BSON::DBRef

This is a subclass of BSON::Document that provides accessors for the collection, id, and database of the DBRef.

BSON::DBRef.new({"$ref" => "collection", "$id" => "id"})
BSON::DBRef.new({"$ref" => "collection", "$id" => "id", "database" => "db"})

bson = {"$ref" => "collection", "$id" => "id"}.to_bson.to_s
loaded = Hash.from_bson(BSON::ByteBuffer.new(bson))
=> {"$ref"=>"collection", "$id"=>"id"}
loaded.class
=> BSON::DBRef

For backwards compatibility with the MongoDB Ruby driver versions 2.17 and earlier, BSON::DBRef also can be constructed using the legacy driver API. This API is deprecated and will be removed in a future version of bson-ruby:

BSON::DBRef.new("collection", BSON::ObjectId('61eeb760a15d5d0f9f1e401d'))
BSON::DBRef.new("collection", BSON::ObjectId('61eeb760a15d5d0f9f1e401d'), "db")

BSON::Document

This is a subclass of Hash that stores all keys as strings, but allows access to them with symbol keys.

BSON::Document[:key, "value"]
BSON::Document.new

bson = {test: 1}.to_bson.to_s
loaded = Hash.from_bson(BSON::ByteBuffer.new(bson))
=> {"test"=>1}
loaded.class
=> BSON::Document

BSON::MaxKey

Represents a value in BSON that will always compare higher to another value.

BSON::MaxKey.new

BSON::MinKey

Represents a value in BSON that will always compare lower to another value.

BSON::MinKey.new

BSON::ObjectId

Represents a 12 byte unique identifier for an object on a given machine.

BSON::ObjectId.new

BSON::Timestamp

Represents a special time with a start and increment value.

BSON::Timestamp.new(5, 30)

BSON::Undefined

Represents a placeholder for a value that was not provided.

BSON::Undefined.new

BSON::Decimal128

Represents a 128-bit decimal-based floating-point value capable of emulating decimal rounding with exact precision.

# Instantiate with a String
BSON::Decimal128.new("1.28")
# Instantiate with a BigDecimal
d = BigDecimal(1.28, 3)
BSON::Decimal128.new(d)

Symbol

The BSON specification defines a symbol type which allows round-tripping Ruby Symbol values (i.e., a Ruby Symbol``is encoded into a BSON symbol and a BSON symbol is decoded into a Ruby ``Symbol). However, since most programming langauges do not have a native symbol type, to promote interoperabilty, MongoDB deprecated the BSON symbol type and encourages strings to be used instead.

Hash.from_bson({foo: 'bar'}.to_bson)
# => {"foo"=>"bar"}
Hash.from_bson({1 => 2}.to_bson)
# => {"1"=>2}

By default, the BSON library encodes Symbol hash values as strings and decodes BSON symbols into Ruby Symbol values:

{foo: :bar}.to_bson.to_s
# => "\x12\x00\x00\x00\x02foo\x00\x04\x00\x00\x00bar\x00\x00"
# 0x02 is the string type
Hash.from_bson(BSON::ByteBuffer.new("\x12\x00\x00\x00\x02foo\x00\x04\x00\x00\x00bar\x00\x00".force_encoding('BINARY')))
# => {"foo"=>"bar"}
# 0x0E is the symbol type
Hash.from_bson(BSON::ByteBuffer.new("\x12\x00\x00\x00\x0Efoo\x00\x04\x00\x00\x00bar\x00\x00".force_encoding('BINARY')))
# => {"foo"=>:bar}

To force encoding of Ruby symbols to BSON symbols, wrap the Ruby symbols in BSON::Symbol::Raw:

{foo: BSON::Symbol::Raw.new(:bar)}.to_bson.to_s
# => "\x12\x00\x00\x00\x0Efoo\x00\x04\x00\x00\x00bar\x00\x00"

Ruby vs MongoDB Regular Expressions

MongoDB server uses Perl-compatible regular expressions implemented using the PCRE library and Ruby regular expressions are implemented using the Onigmo regular expression engine, which is a fork of Oniguruma. The two regular expression implementations generally provide equivalent functionality but have several important syntax differences, as described below.

Unfortunately, there is no simple way to programmatically convert a PCRE regular expression into the equivalent Ruby regular expression, and there are currently no Ruby bindings for PCRE.

BSON::Regexp::Raw Class

Since there is no simple way to programmatically convert a PCRE regular expression into the equivalent Ruby regular expression, bson-ruby provides the BSON::Regexp::Raw class for holding MongoDB/PCRE regular expressions. Instances of this class are called "BSON regular expressions" in this documentation.

Instances of this class can be created using the regular expression text as a string and optional PCRE modifiers:

BSON::Regexp::Raw.new("^b403158")
# => #<BSON::Regexp::Raw:0x000055df63186d78 @pattern="^b403158", @options="">
BSON::Regexp::Raw.new("^Hello.world$", "s")
# => #<BSON::Regexp::Raw:0x000055df6317f028 @pattern="^Hello.world$", @options="s">

The BSON::Regexp module is included in the Ruby Regexp class, such that the BSON:: prefix may be omitted:

Regexp::Raw.new("^b403158")
# => #<BSON::Regexp::Raw:0x000055df63186d78 @pattern="^b403158", @options="">
Regexp::Raw.new("^Hello.world$", "s")
# => #<BSON::Regexp::Raw:0x000055df6317f028 @pattern="^Hello.world$", @options="s">

Regular Expression Conversion

To convert a Ruby regular expression to a BSON regular expression, instantiate a BSON::Regexp::Raw object as follows:

regexp = /^Hello.world/
bson_regexp = BSON::Regexp::Raw.new(regexp.source, regexp.options)
# => #<BSON::Regexp::Raw:0x000055df62e42d60 @pattern="^Hello.world", @options=0>

Note that the BSON::Regexp::Raw constructor accepts both the Ruby numeric options and the PCRE modifier strings.

To convert a BSON regular expression to a Ruby regular expression, call the compile method on the BSON regular expression:

bson_regexp = BSON::Regexp::Raw.new("^hello.world", "s")
bson_regexp.compile
# => /^hello.world/m
bson_regexp = BSON::Regexp::Raw.new("^hello", "")
bson_regexp.compile
# => /^hello.world/
bson_regexp = BSON::Regexp::Raw.new("^hello.world", "m")
bson_regexp.compile
# => /^hello.world/

Note that the s PCRE modifier was converted to the m Ruby modifier in the first example, and the last two examples were converted to the same regular expression even though the original BSON regular expressions had different meanings.

When a BSON regular expression uses the non-portable ^ and $ anchors, its conversion to a Ruby regular expression can change its meaning:

BSON::Regexp::Raw.new("^hello.world", "").compile =~ "42\nhello world"
# => 3

When a Ruby regular expression is converted to a BSON regular expression (for example, to send to the server as part of a query), the BSON regular expression always has the m modifier set reflecting the behavior of ^ and $ anchors in Ruby regular expressions.

Reading and Writing

Both Ruby and BSON regular expressions implement the to_bson method for serialization to BSON:

regexp_ruby = /^b403158/
# => /^b403158/
regexp_ruby.to_bson
# => #<BSON::ByteBuffer:0x007fcf20ab8028>
_.to_s
# => "^b403158\x00m\x00"
regexp_raw = Regexp::Raw.new("^b403158")
# => #<BSON::Regexp::Raw:0x007fcf21808f98 @pattern="^b403158", @options="">
regexp_raw.to_bson
# => #<BSON::ByteBuffer:0x007fcf213622f0>
_.to_s
# => "^b403158\x00\x00"

Both Regexp and BSON::Regexp::Raw classes implement the from_bson class method that deserializes a regular expression from a BSON byte buffer. Methods of both classes return a BSON::Regexp::Raw instance that must be converted to a Ruby regular expression using the compile method as described above.

byte_buffer = BSON::ByteBuffer.new("^b403158\x00\x00")
regex = Regexp.from_bson(byte_buffer)
# => #<BSON::Regexp::Raw:0x000055df63100d40 @pattern="^b403158", @options="">
regex.pattern
# => "^b403158"
regex.options
# => ""
regex.compile
# => /^b403158/