TODO - third_party/libssh2 - Git at Google

 Things TODO
 ===========

 * Fix the numerous malloc+copy operations for sending data, see "Buffering
   Improvements" below for details

 * make sure the windowing code adapts better to slow situations so that it
   doesn't then use as much memory as today. Possibly by an app-controllable
   "Window mode"?

 * Decrease the number of mallocs. Everywhere. Will get easier once the
   buffering improvements have been done.

 * Use SO_NOSIGPIPE for Mac OS/BSD systems where MSG_NOSIGNAL doesn't
   exist/work

 * Extend the test suite to actually test lots of aspects of libssh2

 * Fix all compiler warnings (some can't be done without API changes)

 * Expose error messages sent by the server

 * select() is troublesome with libssh2 when using multiple channels over
   the same session. See "New Transport API" below for more details.

 At next SONAME bump
 ===================

 * stop using #defined macros as part of the official API. The macros should
   either be turned into real functions or discarded from the API.

 * fix the parts of the API where object pointers and function pointers are
   mixed like libssh2_session_callback_set()

 * remove the following functions from the API/ABI

   libssh2_base64_decode()
   libssh2_session_flag()
   libssh2_channel_handle_extended_data()
   libssh2_channel_receive_window_adjust()
   libssh2_poll()
   libssh2_poll_channel_read()
   libssh2_session_startup() (libssh2_session_handshake() is the replacement)
   libssh2_banner_set() (libssh2_session_banner_set() is the repacement)

 * Rename a few function:

   libssh2_hostkey_hash => libssh2_session_hostkey_hash
   libssh2_banner_set => libssh2_session_banner_set

 * change 'int' to 'libssh2_socket_t' in the public API for sockets.

 * Use 'size_t' for string lengths in all functions.

 * Add a comment field to struct libssh2_knownhost.

 * remove the existing libssh2_knownhost_add() function and rename
   libssh2_knownhost_addc to become the new libssh2_knownhost_add instead

 * remove the existing libssh2_scp_send_ex() function and rename
   libssh2_scp_send64 to become the new libssh2_scp_send instead.

 * remove the existing libssh2_knownhost_check() functin and rename
   libssh2_knownhost_checkp() to become the new libssh2_knownhost_check instead

 Buffering Improvements
 ======================

 transport_write

   - If this function gets called with a total packet size that is larger than
   32K, it should create more than one SSH packet so that it keeps the largest
   one below 32K

 sftp_write

   - should not copy/allocate anything for the data, only create a header chunk
   and pass on the payload data to channel_write "pointed to"

 New Transport API
 =================

 THE PROBLEM

 The problem in a nutshell is that when an application opens up multiple
 channels over a single session, those are all using the same socket. If the
 application is then using select() to wait for traffic (like any sensible app
 does) and wants to act on the data when select() tells there is something to
 for example read, what does an application do?

 With our current API, you have to loop over all the channels and read from
 them to see if they have data. This effectively makes blocking reads
 impossible. If the app has many channels in a setup like this, it even becomes
 slow. (The original API had the libssh2_poll_channel_read() and libssh2_poll()
 to somewhat overcome this hurdle, but they too have pretty much the same
 problems plus a few others.)

 Traffic in the other direction is similarly limited: the app has to try
 sending to all channels, even though some of them may very well not accept any
 data at that point.

 A SOLUTION

 I suggest we introduce two new helper functions:

  libssh2_transport_read()

  - Read "a bunch" of data from the given socket and returns information to the
    app about what channels that are now readable (ie they will not block when
    read from). The function can be called over and over and it will repeatedly
    return info about what channels that are readable at that moment.

  libssh2_transport_write()

  - Returns information about what channels that are writable, in the sense
    that they have windows set from the remote side that allows data to get
    sent. Writing to one of those channels will not block. Of course, the
    underlying socket may only accept a certain amount of data, so at the first
    short return, nothing more should be attempted to get sent until select()
    (or equivalent) has been used on the master socket again.

 I haven't yet figured out a sensible API for how these functions should return
 that info, but if we agree on the general principles I guess we can work that
 out.

 VOLUNTARY

   I wanted to mention that these two helper functions would not be mandatory
   in any way. They would just be there for those who want them, and existing
   programs can remain using the old functions only if they prefer to.

 New SFTP API
 ============

 PURPOSE

  Provide API functions that explicitly tells at once that a (full) SFTP file
  transfer is wanted, to allow libssh2 to leverage on that knowledge to speed
  up things internally. It can for example do read ahead, buffer writes (merge
  small writes into larger chunks), better tune the SSH window and more. This
  sort of API is already provided for SCP transfers.

 API

  New functions:

     LIBSSH2_SFTP_HANDLE *libssh2_sftp_send(SFTP_SESSION *sftp,
                                            uint64_t filesize,
                                            char *remote_path,
                                            size_t remote_path_len,
                                            long mode);

  Tell libssh2 that a local file with a given size is about to get sent to
  the SFTP server.

     LIBSSH2_SFTP_HANDLE *libssh2_sftp_recv();

  Tell libssh2 that a remote file is requested to get downloaded from the SFTP
  server.

  Only the setup of the file transfer is different from an application's point
  of view. Depending on direction of the transfer(s), the following already
  existing functions should then be used until the transfer is complete:

  libssh2_sftp_read()
  libssh2_sftp_write()

 HOW TO USE

  1. Setup the transfer using one of the two new functions.

  2. Loop through the reading or writing of data.

  3. Cleanup the transfer
	Things TODO
	===========

	* Fix the numerous malloc+copy operations for sending data, see "Buffering
	Improvements" below for details

	* make sure the windowing code adapts better to slow situations so that it
	doesn't then use as much memory as today. Possibly by an app-controllable
	"Window mode"?

	* Decrease the number of mallocs. Everywhere. Will get easier once the
	buffering improvements have been done.

	* Use SO_NOSIGPIPE for Mac OS/BSD systems where MSG_NOSIGNAL doesn't
	exist/work

	* Extend the test suite to actually test lots of aspects of libssh2

	* Fix all compiler warnings (some can't be done without API changes)

	* Expose error messages sent by the server

	* select() is troublesome with libssh2 when using multiple channels over
	the same session. See "New Transport API" below for more details.

	At next SONAME bump
	===================

	* stop using #defined macros as part of the official API. The macros should
	either be turned into real functions or discarded from the API.

	* fix the parts of the API where object pointers and function pointers are
	mixed like libssh2_session_callback_set()

	* remove the following functions from the API/ABI

	libssh2_base64_decode()
	libssh2_session_flag()
	libssh2_channel_handle_extended_data()
	libssh2_channel_receive_window_adjust()
	libssh2_poll()
	libssh2_poll_channel_read()
	libssh2_session_startup() (libssh2_session_handshake() is the replacement)
	libssh2_banner_set() (libssh2_session_banner_set() is the repacement)

	* Rename a few function:

	libssh2_hostkey_hash => libssh2_session_hostkey_hash
	libssh2_banner_set => libssh2_session_banner_set

	* change 'int' to 'libssh2_socket_t' in the public API for sockets.

	* Use 'size_t' for string lengths in all functions.

	* Add a comment field to struct libssh2_knownhost.

	* remove the existing libssh2_knownhost_add() function and rename
	libssh2_knownhost_addc to become the new libssh2_knownhost_add instead

	* remove the existing libssh2_scp_send_ex() function and rename
	libssh2_scp_send64 to become the new libssh2_scp_send instead.

	* remove the existing libssh2_knownhost_check() functin and rename
	libssh2_knownhost_checkp() to become the new libssh2_knownhost_check instead

	Buffering Improvements
	======================

	transport_write

	- If this function gets called with a total packet size that is larger than
	32K, it should create more than one SSH packet so that it keeps the largest
	one below 32K

	sftp_write

	- should not copy/allocate anything for the data, only create a header chunk
	and pass on the payload data to channel_write "pointed to"

	New Transport API
	=================

	THE PROBLEM

	The problem in a nutshell is that when an application opens up multiple
	channels over a single session, those are all using the same socket. If the
	application is then using select() to wait for traffic (like any sensible app
	does) and wants to act on the data when select() tells there is something to
	for example read, what does an application do?

	With our current API, you have to loop over all the channels and read from
	them to see if they have data. This effectively makes blocking reads
	impossible. If the app has many channels in a setup like this, it even becomes
	slow. (The original API had the libssh2_poll_channel_read() and libssh2_poll()
	to somewhat overcome this hurdle, but they too have pretty much the same
	problems plus a few others.)

	Traffic in the other direction is similarly limited: the app has to try
	sending to all channels, even though some of them may very well not accept any
	data at that point.

	A SOLUTION

	I suggest we introduce two new helper functions:

	libssh2_transport_read()

	- Read "a bunch" of data from the given socket and returns information to the
	app about what channels that are now readable (ie they will not block when
	read from). The function can be called over and over and it will repeatedly
	return info about what channels that are readable at that moment.

	libssh2_transport_write()

	- Returns information about what channels that are writable, in the sense
	that they have windows set from the remote side that allows data to get
	sent. Writing to one of those channels will not block. Of course, the
	underlying socket may only accept a certain amount of data, so at the first
	short return, nothing more should be attempted to get sent until select()
	(or equivalent) has been used on the master socket again.

	I haven't yet figured out a sensible API for how these functions should return
	that info, but if we agree on the general principles I guess we can work that
	out.

	VOLUNTARY

	I wanted to mention that these two helper functions would not be mandatory
	in any way. They would just be there for those who want them, and existing
	programs can remain using the old functions only if they prefer to.

	New SFTP API
	============

	PURPOSE

	Provide API functions that explicitly tells at once that a (full) SFTP file
	transfer is wanted, to allow libssh2 to leverage on that knowledge to speed
	up things internally. It can for example do read ahead, buffer writes (merge
	small writes into larger chunks), better tune the SSH window and more. This
	sort of API is already provided for SCP transfers.

	API

	New functions:

	LIBSSH2_SFTP_HANDLE libssh2_sftp_send(SFTP_SESSION sftp,
	uint64_t filesize,
	char *remote_path,
	size_t remote_path_len,
	long mode);

	Tell libssh2 that a local file with a given size is about to get sent to
	the SFTP server.

	LIBSSH2_SFTP_HANDLE *libssh2_sftp_recv();

	Tell libssh2 that a remote file is requested to get downloaded from the SFTP
	server.

	Only the setup of the file transfer is different from an application's point
	of view. Depending on direction of the transfer(s), the following already
	existing functions should then be used until the transfer is complete:

	libssh2_sftp_read()
	libssh2_sftp_write()

	HOW TO USE

	1. Setup the transfer using one of the two new functions.

	2. Loop through the reading or writing of data.

	3. Cleanup the transfer