How do I send raw binary data via HTTP in node.js?

Question

From a node.js back end, I need to send an HTTP message to a REST endpoint. The endpoint requires some parameters that it will expect to find in the HTTP message. Some of the parameters are simple enough, just requiring a number or a string as an argument. But one of the parameters is to be "the raw binary file content being uploaded" and this has puzzled me. As far as I understand, the parameters need to be gathered together into a string to put in the body of the HTTP request; How do I add raw binary data to a string? Obviously, for it to be in the string, it cannot be raw binary data; it needs to be encoded into characters.

The endpoint in question is the Twitter media upload API. The "raw binary data" parameter is called media. Below is an incomplete code snippet showing the basic gist of what I've tried. Specifically, the line where I build the requestBody string. I don't believe it is anywhere near correct, because the endpoint is returning a "bad request" message.

var https = require("https");

var base64ImageData = /* (some base 64 string here) */;

var options = {
    host: "api.twitter.com",
    path: "/1.1/media/upload.json",
    method: "POST",
    headers: {
        "Content-Type": "multipart/form-data"
    }
};
var request = https.request(options, function(response) {});
var requestBody = "media_id=18283918294&media=" + Buffer.from(base64ImageData, "base64").toString("binary");
request.write(requestBody);
request.end();

Also worth noting, Twitter themselves note the following extremely confusing statement:

"When posting base64 encoded images, be sure to set the “Content-Transfer-Encoding: base64” on the image part of the message."

Source: https://developer.twitter.com/en/docs/media/upload-media/uploading-media/media-best-practices

That might be part of the answer to my question, but what I don't understand is: How do I apply different headers to different parts of the HTTP message? Because apparently, the image data needs to have a Content-Transfer-Encoding header of "base64" while the rest of the HTTP message does not...

ceejayoz · Accepted Answer · 2019-05-29 02:18:11Z

3

How do I apply different headers to different parts of the HTTP message?

This is the point of the multipart/form-data content type. A multi-part message looks like this:

Content-Type: multipart/form-data; boundary=---foo---

---foo---
Content-Disposition: form-data; name="datafile1"; filename="r.gif"
Content-Transfer-Encoding: base64
Content-Type: image/gif

// data goes here
---foo---
Content-Disposition: form-data; name="datafile2"; filename="g.png"
Content-Transfer-Encoding: base64
Content-Type: image/png

// another file's data goes here
---foo---

You probably don't want to put all this together yourself. There are a bunch of good libraries for putting together complex POSTs. For example: https://www.npmjs.com/package/form-data

answered May 29, 2019 at 2:18

ceejayoz

181k42 gold badges311 silver badges382 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

CircuitScholar Over a year ago

Thank you, but this still leaves me baffled by the specific case of Twitter. Their documentation strongly implies the file data is just supposed to be provided in sort of a URL parameter format. developer.twitter.com/en/docs/media/upload-media/api-reference/…. See the provided example there: command=APPEND&media_id=123&segment_index=2&media_data=123 Can you speculate as to why that may be the case? Also just to be clear, in your example of a multi-part message, does all of that go in the body? Does the boundary=--foo-- go in the Content-Type header?

Jake Holzinger Over a year ago

@CircuitScholar those are query parameters, they are not part of the request body. When that format is used in the request body the content type is application/x-www-form-urlencoded.

ceejayoz Over a year ago

@CircuitScholar Twitter's docs have always been a bit shitty. Stop assuming implications - chances are if something's weird/nonsensical in their docs, it's just Twitter being Twitter. You don't send large multi-megabyte file data as a URL param.

ceejayoz Over a year ago

(Side note: See the step-by-step guide at developer.twitter.com/en/docs/media/upload-media/… for chunked uploads, and make use of this bit: "To see full headers of the requests and responses when using twurl, use the -t option to enable trace mode.")

Collectives™ on Stack Overflow

How do I send raw binary data via HTTP in node.js?

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related