| <!-- HTML header for doxygen 1.8.6--> |
| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/> |
| <meta http-equiv="X-UA-Compatible" content="IE=9"/> |
| <meta name="generator" content="Doxygen 1.9.1"/> |
| <title>FlatBuffers: FlexBuffers</title> |
| <link href="tabs.css" rel="stylesheet" type="text/css"/> |
| <script type="text/javascript" src="jquery.js"></script> |
| <script type="text/javascript" src="dynsections.js"></script> |
| <link href="navtree.css" rel="stylesheet" type="text/css"/> |
| <script type="text/javascript" src="resize.js"></script> |
| <script type="text/javascript" src="navtreedata.js"></script> |
| <script type="text/javascript" src="navtree.js"></script> |
| <link href="search/search.css" rel="stylesheet" type="text/css"/> |
| <script type="text/javascript" src="search/searchdata.js"></script> |
| <script type="text/javascript" src="search/search.js"></script> |
| <link href="doxygen.css" rel="stylesheet" type="text/css" /> |
| <link href="https://fonts.googleapis.com/css?family=Roboto:300,400,400italic,500,500italic,700,700italic|Roboto+Mono:400,700" rel="stylesheet"> |
| <link href="style.css" rel="stylesheet" type="text/css"/> |
| </head> |
| <body> |
| <div id="top"><!-- do not remove this div, it is closed by doxygen! --> |
| <div id="titlearea" style="height: 110px;"> |
| <table cellspacing="0" cellpadding="0"> |
| <tbody> |
| <tr style="height: 56px;"> |
| <td id="commonprojectlogo"> |
| <img alt="Logo" src="fpl_logo_small.png"/> |
| </td> |
| <td style="padding-left: 0.5em;"> |
| <div id="projectname">FlatBuffers |
| </div> |
| <div style="font-size:12px;"> |
| An open source project by <a href="https://developers.google.com/games/#Tools">FPL</a>. |
| </div> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <!-- end header part --> |
| <!-- Generated by Doxygen 1.9.1 --> |
| <script type="text/javascript"> |
| /* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2 */ |
| var searchBox = new SearchBox("searchBox", "search",false,'Search','.html'); |
| /* @license-end */ |
| </script> |
| <script type="text/javascript" src="menudata.js"></script> |
| <script type="text/javascript" src="menu.js"></script> |
| <script type="text/javascript"> |
| /* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2 */ |
| $(function() { |
| initMenu('',true,false,'search.php','Search'); |
| $(document).ready(function() { init_search(); }); |
| }); |
| /* @license-end */</script> |
| <div id="main-nav"></div> |
| </div><!-- top --> |
| <div id="side-nav" class="ui-resizable side-nav-resizable"> |
| <div id="nav-tree"> |
| <div id="nav-tree-contents"> |
| <div id="nav-sync" class="sync"></div> |
| </div> |
| </div> |
| <div id="splitbar" style="-moz-user-select:none;" |
| class="ui-resizable-handle"> |
| </div> |
| </div> |
| <script type="text/javascript"> |
| /* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2 */ |
| $(document).ready(function(){initNavTree('flexbuffers.html',''); initResizable(); }); |
| /* @license-end */ |
| </script> |
| <div id="doc-content"> |
| <!-- window showing the filter options --> |
| <div id="MSearchSelectWindow" |
| onmouseover="return searchBox.OnSearchSelectShow()" |
| onmouseout="return searchBox.OnSearchSelectHide()" |
| onkeydown="return searchBox.OnSearchSelectKey(event)"> |
| </div> |
| |
| <!-- iframe showing the search results (closed by default) --> |
| <div id="MSearchResultsWindow"> |
| <iframe src="javascript:void(0)" frameborder="0" |
| name="MSearchResults" id="MSearchResults"> |
| </iframe> |
| </div> |
| |
| <div class="PageDoc"><div class="header"> |
| <div class="headertitle"> |
| <div class="title">FlexBuffers </div> </div> |
| </div><!--header--> |
| <div class="contents"> |
| <div class="textblock"><p><a class="anchor" id="md_FlexBuffers"></a></p> |
| <p>FlatBuffers was designed around schemas, because when you want maximum performance and data consistency, strong typing is helpful.</p> |
| <p>There are however times when you want to store data that doesn't fit a schema, because you can't know ahead of time what all needs to be stored.</p> |
| <p>For this, FlatBuffers has a dedicated format, called FlexBuffers. This is a binary format that can be used in conjunction with FlatBuffers (by storing a part of a buffer in FlexBuffers format), or also as its own independent serialization format.</p> |
| <p>While it loses the strong typing, you retain the most unique advantage FlatBuffers has over other serialization formats (schema-based or not): FlexBuffers can also be accessed without parsing / copying / object allocation. This is a huge win in efficiency / memory friendly-ness, and allows unique use cases such as mmap-ing large amounts of free-form data.</p> |
| <p>FlexBuffers' design and implementation allows for a very compact encoding, combining automatic pooling of strings with automatic sizing of containers to their smallest possible representation (8/16/32/64 bits). Many values and offsets can be encoded in just 8 bits. While a schema-less representation is usually more bulky because of the need to be self-descriptive, FlexBuffers generates smaller binaries for many cases than regular FlatBuffers.</p> |
| <p>FlexBuffers is still slower than regular FlatBuffers though, so we recommend to only use it if you need it.</p> |
| <h1><a class="anchor" id="autotoc_md150"></a> |
| Usage in C++</h1> |
| <p>Include the header <code>flexbuffers.h</code>, which in turn depends on <code>flatbuffers.h</code> and <code>util.h</code>.</p> |
| <p>To create a buffer:</p> |
| <div class="fragment"><div class="line">flexbuffers::Builder fbb;</div> |
| <div class="line">fbb.Int(13);</div> |
| <div class="line">fbb.Finish();</div> |
| </div><!-- fragment --><p>You create any value, followed by <code>Finish</code>. Unlike FlatBuffers which requires the root value to be a table, here any value can be the root, including a lonely int value.</p> |
| <p>You can now access the <code>std::vector<uint8_t></code> that contains the encoded value as <code>fbb.GetBuffer()</code>. Write it, send it, or store it in a parent FlatBuffer. In this case, the buffer is just 3 bytes in size.</p> |
| <p>To read this value back, you could just say:</p> |
| <div class="fragment"><div class="line"><span class="keyword">auto</span> root = flexbuffers::GetRoot(my_buffer);</div> |
| <div class="line">int64_t i = root.AsInt64();</div> |
| </div><!-- fragment --><p>FlexBuffers stores ints only as big as needed, so it doesn't differentiate between different sizes of ints. You can ask for the 64 bit version, regardless of what you put in. In fact, since you demand to read the root as an int, if you supply a buffer that actually contains a float, or a string with numbers in it, it will convert it for you on the fly as well, or return 0 if it can't. If instead you actually want to know what is inside the buffer before you access it, you can call <code>root.GetType()</code> or <code>root.IsInt()</code> etc.</p> |
| <p>Here's a slightly more complex value you could write instead of <code>fbb.Int</code> above:</p> |
| <div class="fragment"><div class="line">fbb.Map([&]() {</div> |
| <div class="line"> fbb.Vector(<span class="stringliteral">"vec"</span>, [&]() {</div> |
| <div class="line"> fbb.Int(-100);</div> |
| <div class="line"> fbb.String(<span class="stringliteral">"Fred"</span>);</div> |
| <div class="line"> fbb.IndirectFloat(4.0f);</div> |
| <div class="line"> });</div> |
| <div class="line"> fbb.UInt(<span class="stringliteral">"foo"</span>, 100);</div> |
| <div class="line">});</div> |
| </div><!-- fragment --><p>This stores the equivalent of the JSON value <code>{ vec: [ -100, "Fred", 4.0 ], foo: 100 }</code>. The root is a dictionary that has just two key-value pairs, with keys <code>vec</code> and <code>foo</code>. Unlike FlatBuffers, it actually has to store these keys in the buffer (which it does only once if you store multiple such objects, by pooling key values), but also unlike FlatBuffers it has no restriction on the keys (fields) that you use.</p> |
| <p>The map constructor uses a C++11 Lambda to group its children, but you can also use more conventional start/end calls if you prefer.</p> |
| <p>The first value in the map is a vector. You'll notice that unlike FlatBuffers, you can use mixed types. There is also a <code>TypedVector</code> variant that only allows a single type, and uses a bit less memory.</p> |
| <p><code>IndirectFloat</code> is an interesting feature that allows you to store values by offset rather than inline. Though that doesn't make any visible change to the user, the consequence is that large values (especially doubles or 64 bit ints) that occur more than once can be shared (see ReuseValue). Another use case is inside of vectors, where the largest element makes up the size of all elements (e.g. a single double forces all elements to 64bit), so storing a lot of small integers together with a double is more efficient if the double is indirect.</p> |
| <p>Accessing it:</p> |
| <div class="fragment"><div class="line"><span class="keyword">auto</span> map = flexbuffers::GetRoot(my_buffer).AsMap();</div> |
| <div class="line">map.size(); <span class="comment">// 2</span></div> |
| <div class="line"><span class="keyword">auto</span> vec = map[<span class="stringliteral">"vec"</span>].AsVector();</div> |
| <div class="line">vec.size(); <span class="comment">// 3</span></div> |
| <div class="line">vec[0].AsInt64(); <span class="comment">// -100;</span></div> |
| <div class="line">vec[1].AsString().c_str(); <span class="comment">// "Fred";</span></div> |
| <div class="line">vec[1].AsInt64(); <span class="comment">// 0 (Number parsing failed).</span></div> |
| <div class="line">vec[2].AsDouble(); <span class="comment">// 4.0</span></div> |
| <div class="line">vec[2].AsString().IsTheEmptyString(); <span class="comment">// true (Wrong Type).</span></div> |
| <div class="line">vec[2].AsString().c_str(); <span class="comment">// "" (This still works though).</span></div> |
| <div class="line">vec[2].ToString().c_str(); <span class="comment">// "4" (Or have it converted).</span></div> |
| <div class="line">map[<span class="stringliteral">"foo"</span>].AsUInt8(); <span class="comment">// 100</span></div> |
| <div class="line">map[<span class="stringliteral">"unknown"</span>].IsNull(); <span class="comment">// true</span></div> |
| </div><!-- fragment --><h1><a class="anchor" id="autotoc_md151"></a> |
| Usage in Java</h1> |
| <p>Java implementation follows the C++ one, closely.</p> |
| <p>For creating the equivalent of the same JSON <code>{ vec: [ -100, "Fred", 4.0 ], foo: 100 }</code>, one could use the following code:</p> |
| <div class="fragment"><div class="line">FlexBuffersBuilder builder = <span class="keyword">new</span> FlexBuffersBuilder(ByteBuffer.allocate(512),</div> |
| <div class="line"> FlexBuffersBuilder.BUILDER_FLAG_SHARE_KEYS_AND_STRINGS);</div> |
| <div class="line"><span class="keywordtype">int</span> smap = builder.startMap();</div> |
| <div class="line"><span class="keywordtype">int</span> svec = builder.startVector();</div> |
| <div class="line">builder.putInt(-100);</div> |
| <div class="line">builder.putString(<span class="stringliteral">"Fred"</span>);</div> |
| <div class="line">builder.putFloat(4.0);</div> |
| <div class="line">builder.endVector(<span class="stringliteral">"vec"</span>, svec, <span class="keyword">false</span>, <span class="keyword">false</span>);</div> |
| <div class="line">builder.putInt(<span class="stringliteral">"foo"</span>, 100);</div> |
| <div class="line">builder.endMap(<span class="keyword">null</span>, smap);</div> |
| <div class="line">ByteBuffer bb = builder.finish();</div> |
| </div><!-- fragment --><p>Similarly, to read the data, just:</p> |
| <div class="fragment"><div class="line">FlexBuffers.Map map = FlexBuffers.getRoot(bb).asMap();</div> |
| <div class="line">map.size(); <span class="comment">// 2</span></div> |
| <div class="line">FlexBuffers.Vector vec = map.get(<span class="stringliteral">"vec"</span>).asVector();</div> |
| <div class="line">vec.size(); <span class="comment">// 3</span></div> |
| <div class="line">vec.get(0).asLong(); <span class="comment">// -100;</span></div> |
| <div class="line">vec.get(1).asString(); <span class="comment">// "Fred";</span></div> |
| <div class="line">vec.get(1).asLong(); <span class="comment">// 0 (Number parsing failed).</span></div> |
| <div class="line">vec.get(2).asFloat(); <span class="comment">// 4.0</span></div> |
| <div class="line">vec.get(2).asString().isEmpty(); <span class="comment">// true (Wrong Type).</span></div> |
| <div class="line">vec.get(2).asString(); <span class="comment">// "" (This still works though).</span></div> |
| <div class="line">vec.get(2).toString(); <span class="comment">// "4.0" (Or have it converted).</span></div> |
| <div class="line">map.get(<span class="stringliteral">"foo"</span>).asUInt(); <span class="comment">// 100</span></div> |
| <div class="line">map.get(<span class="stringliteral">"unknown"</span>).isNull(); <span class="comment">// true</span></div> |
| </div><!-- fragment --><h1><a class="anchor" id="autotoc_md152"></a> |
| Binary encoding</h1> |
| <p>A description of how FlexBuffers are encoded is in the <a class="el" href="flatbuffers_internals.html">internals</a> document.</p> |
| <h1><a class="anchor" id="autotoc_md153"></a> |
| Nesting inside a FlatBuffer</h1> |
| <p>You can mark a field as containing a FlexBuffer, e.g. </p><pre class="fragment">a:[ubyte] (flexbuffer); |
| </pre><p> A special accessor will be generated that allows you to access the root value directly, e.g. <code>a_flexbuffer_root().AsInt64()</code>.</p> |
| <h1><a class="anchor" id="autotoc_md154"></a> |
| Efficiency tips</h1> |
| <ul> |
| <li>Vectors generally are a lot more efficient than maps, so prefer them over maps when possible for small objects. Instead of a map with keys <code>x</code>, <code>y</code> and <code>z</code>, use a vector. Better yet, use a typed vector. Or even better, use a fixed size typed vector.</li> |
| <li>Maps are backwards compatible with vectors, and can be iterated as such. You can iterate either just the values (<code>map.Values()</code>), or in parallel with the keys vector (<code>map.Keys()</code>). If you intend to access most or all elements, this is faster than looking up each element by key, since that involves a binary search of the key vector.</li> |
| <li>When possible, don't mix values that require a big bit width (such as double) in a large vector of smaller values, since all elements will take on this width. Use <code>IndirectDouble</code> when this is a possibility. Note that integers automatically use the smallest width possible, i.e. if you ask to serialize an int64_t whose value is actually small, you will use less bits. Doubles are represented as floats whenever possible losslessly, but this is only possible for few values. Since nested vectors/maps are stored over offsets, they typically don't affect the vector width.</li> |
| <li>To store large arrays of byte data, use a blob. If you'd use a typed vector, the bit width of the size field may make it use more space than expected, and may not be compatible with <code>memcpy</code>. Similarly, large arrays of (u)int16_t may be better off stored as a binary blob if their size could exceed 64k elements. Construction and use are otherwise similar to strings. </li> |
| </ul> |
| </div></div><!-- contents --> |
| </div><!-- PageDoc --> |
| </div><!-- doc-content --> |
| <!-- Google Analytics --> |
| <script> |
| (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ |
| (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), |
| m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) |
| })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); |
| ga('create', 'UA-49880327-7', 'auto'); |
| ga('send', 'pageview'); |
| </script> |
| </body> |
| </html> |
