How to pass image frames camera to a function in wasm (C++)? - javascript

I'm trying to build a C++ function and compile it to WASM using Emscripten.
What this function will do is receive an image and do some process on it and return a result.
My first POC was successful, the user upload image using file input and I pass the data of the image using FileReader API:
const fileReader = new FileReader();
fileReader.onload = (event) => {
const uint8Arr = new Uint8Array(event.target.result);
passToWASM(event.target.result);
};
fileReader.readAsArrayBuffer(file); // I got this `file` from `change` event of the file input.
But when I implemented the camera feed and started to get frames to pass it to WASM, I started to get exceptions in C++ side, and here's the JS implementation:
let imageData = canvasCtx.getImageData(0, 0, videoWidth, videoHeight);
var data=imageData.data.buffer;
var uint8Arr = new Uint8Array(data);
passToWASM(uint8Arr);
This one throws an exception in C++ side.
Now passToWASM implementation is:
function passToWASM(uint8ArrData) {
// copying the uint8ArrData to the heap
const numBytes = uint8ArrData.length * uint8ArrData.BYTES_PER_ELEMENT;
const dataPtr = Module._malloc(numBytes);
const dataOnHeap = new Uint8Array(Module.HEAPU8.buffer, dataPtr, numBytes);
dataOnHeap.set(uint8ArrData);
// calling the WASM function
const res = Module._myWASMFunc(dataOnHeap.byteOffset, uint8ArrData.length);
}
While the C++ implementation will be something like this:
void EMSCRIPTEN_KEEPALIVE checkImageQuality(uint8_t* buffer, size_t size) {
// I'm using OpenCV in C++ to process the image data
// So I read the data of the image
cv::Mat raw_data = cv::Mat(1, size, CV_8UC1, buffer);
// Then I convert it
cv::Mat img_data = cv::imdecode(raw_data, cv::IMREAD_COLOR | cv::IMREAD_IGNORE_ORIENTATION);
// in one of the following steps I'm using cvtColor function which causes the issue for some reason
}
The exception I'm getting because of the camera implementation says:
OpenCV(4.1.0-dev) ../modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
What is the difference between using file input and getting the data to pass it, and getting the data from a canvas as long as both of them are convert it to Uint8Array

I found a solution for this (maybe suits my case only).
When you're trying to get an image data from canvas you get it as 4 channels (e.g: PNG), and depending on your image processing code you need to deal with it.
My code was considering that the image should be 3 channels (e.g: jpeg) so I had to convert it using this code:
canvasBuffer.toBlob(function (blob) {
passToWASM(blob);
},'image/jpeg');

Related

Unable to make an Image Blob JSON Serializable

JIRA Ticket created due to base64encode failure: https://jira.appcelerator.org/browse/TC-5876
My Current CFG:
Titanium SDK 5.1.2.GA
Testing on an iPhone iOS 9.1
I'm stuck in a problem in a project for a client that requires images took on device (using the camera) to be sent to a WebService and afterwards be seen on any device using the app (both Android and iOS devices).
Titanium provides a Ti.Blob object (event.media) after taking a picture (which is not JSON serializable) and I need somehow to send this to the server. The server responds always a JSON object, thus this Blob must be somehow JSON serializable.
I've tried many ways without success:
1 - Base64Encode the Blob
var base64blob = Ti.Utils.base64encode(event.media);
Doesn't work, it stucks the app and throws a ASL exceeded maximum size error. I imagine that the image is too large to be base64encoded.
2 - Read the Blob into a Buffer
var blobStream = Ti.Stream.createStream({ source: event.media, mode: Ti.Stream.MODE_READ });
var buffer = Ti.createBuffer({ length: event.media.length });
var bytes = blobStream.read(buffer);
It works but I have no idea how can I transform this buffer holding the image contents into something that the server can return in a JSON object and later be transformed into an Image Blob again.
The server can't manage Ti.Blob objects or Ti.Buffer objects because, first of all, they are Titanium objects and the server is C# based, and second due to Ti.Blob and Ti.Buffer aren't JSON serializable, thus the JSON return doesn't work.
What I need is basically described in the imaginary example below:
var imageBlob = event.media;
var JSONSerializableImg = imageBlob.toJSON();
sendImageToServer(JSONSerializableImg);
var imgFromServer = getImageFromServer();
var imageBlob = imgFromServer.toBlob();
var imgView = createImageView({
image: imageBlob
});
I hope someone can help me with any conversion method possible.
Thank's
OK,
This is what I think you have to do. Looking at the API, this is very doable.
1: You need to create an object Server side that will hold the BLOB.
public class BlobContainer
{
public string fileName{get;set;}
//... (Other properties)
public byte[] data {get;set;}
}
2: Convert the important information from the BLOB into a binary array and send to server.
var blobStream = Ti.Stream.createStream({ source: myBlob, mode: Ti.Stream.MODE_READ });
var newBuffer = Ti.createBuffer({ length: myBlob.length });
var bytes = blobStream.read(newBuffer);
3: Then send the byte data to the server through Ajax requests. Be mindful of how big your array is that you are sending. It might be advantageous to split the array up and combine it on the other side (Might not be necessary):
var dataObjects: [
{ id: 1, data: [BYTE_DATA_PART] },
{ id: 2, data: [BYTE_DATA_PART] }...
]
$.each(dataObjects, function(i,a) {
$.ajax({ url: "BLA", data: JSON.stringify(a), dataType: "json", type: "POST",
success: function() { //CONTINUE\\ },
error: function() {alert("ERROR BRO"})
});
});
4: Then server side get each request in your little blob container, store in a session object or cache object and once you have N out of N, piece it all together and store that sucker in the Database.
5: Retrieve the stuff in the reverse order. Just remember that it is stored as byte[] data. You may have to fuddle with it and store it as a string because of the way the TI buffer creates bytes and the way c# interprets bytes. Best thing is trial and error. Once you have all the pieces back on the client.
var newBuffer = Ti.Stream.read(data, 0, data.length);
var newBlob = newBuffer.toBlob();
To send and receive binary data to and from a server it's best to use Ti.Network.HTTPClient which can send and receive binary data.
There's a guide on uploading and downloading files here:
http://docs.appcelerator.com/platform/latest/#!/guide/File_Uploads_and_Downloads
JSON isn't designed to carry binary data, although base64encoded binary data should work. This is what Ti.Utils.base64encode() indeed is for. If you believe the "ASL exceeded maximum size error" you get shouldn't happen, please create a ticket on Appcelerator JIRA
I solved this issue by creating a separate method on server-side specially to upload photos. I followed the link below for server side:
PHP code
In Titanium I had to set XHR's header like this:
this.xhr.setRequestHeader("ContentType", "image/png");
this.xhr.setRequestHeader('enctype', 'multipart/form-data');
That's it!
Thank's for all the answers.
I've used the following method before (not in Titanium, but another web-based mobile app platform).
function convertToDataURLviaCanvas(url, callback, outputFormat){
var img = new Image();
img.crossOrigin = 'Anonymous';
img.onload = function(){
var canvas = document.createElement('CANVAS');
var ctx = canvas.getContext('2d');
var dataURL;
canvas.height = this.height;
canvas.width = this.width;
ctx.drawImage(this, 0, 0);
dataURL = canvas.toDataURL(outputFormat);
callback(dataURL);
canvas = null;
};
img.src = url;
}
convertToDataURLviaCanvas('http://example.com/image.png', function(base64Img){
// Base64DataURL
});
I used this to send the base64 encoded image string as JSON to my backend server. Then re-encoded the image on the server. It worked for me, but the base64 encoded string is huge.

File format of the VTK file to be used as input for XTK

I have a .vtk file at remote server. I am accessing it via http (I verified that the url of the file is correct, in fact I can download it from the browser). I've also tried to remove the blank spaces from the file as suggested in xtk volume rendering with .vtk file created from matlab.
But i get an exception
Unable to get property 'length' of undefined or null reference.
Can you please guide me the .vtk file format that should be used in order to make it work smoothly ?
I am following the xtk tutorial http://jsfiddle.net/gh/get/toolkit/edge/xtk/lessons/tree/master/05/#run, below is my code
window.onload = function () {
// create and initialize a 3D renderer
var r = new X.renderer3D();
r.init();
// create a new X.mesh
var skull = new X.mesh();
// .. and associate the .vtk file to it
skull.file = 'http://localhost/startup/bunnycheck.vtk'; //using other .vtk file
//skull.file = 'http://x.babymri.org/?skull.vtk';
// .. make it transparent
skull.opacity = 0.5;
// .. add the mesh
r.add(skull);
// re-position the camera to face the skull
r.camera.position = [0, 400, 0];
// animate..
r.onRender = function () {
r.camera.rotate([1,0]);
};
r.render();
};
I just got a similar exception, the problem was that the file was a binary vtk file, and it should have been ASCII .

How to read webgl GL.bufferData in javascript

i want to read back the data stored in the GL.bufferData array in javascript.
Here is my code
var TRIANGLE_VERTEX = geometryNode["triangle_buffer"];
GL.bindBuffer(GL.ARRAY_BUFFER, TRIANGLE_VERTEX);
GL.bufferData(GL.ARRAY_BUFFER,new Float32Array(vertices),GL.STATIC_DRAW);
is it possible in webgl to read back the bufferdata in GPU?
if possible then please explain me with a sample code.
How to know the memory size(filled and free) of the Gpu in webgl at run time and how to debug the shader code and data in GPU in webgl.
It is not directly possible to read the data back in WebGL1. (see below for WebGL2). This is limitation of OpenGL ES 2.0 on which WebGL is based.
There are some workarounds:
You could try to render that data to a texture then use readPixels to read the data.
You'd have to encode the data into bytes in your shader because readPixels in WebGL can only read bytes
You can wrap your WebGL to store the data yourself something like
var buffers = {};
var nextId = 1;
var targets = {};
function copyBuffer(buffer) {
// make a Uint8 view of buffer in case it's not already
var view = new Uint8Buffer(buffer.buffer);
// now copy it
return new UintBuffer(view);
}
gl.bindBuffer = function(oldBindBufferFn) {
return function(target, buffer) {
targets[target] = new Uint8Buffer(buffer.buffer);
oldBindBufferFn(target, buffer);
};
}(gl.bindBuffer.bind(gl));
gl.bufferData = function(oldBufferDataFn) {
return function(target, data, hint) {
var buffer = targets[target];
if (!buffer.id) {
buffer.id = nextId++;
}
buffers[buffer.id] = copyBuffer(data);
oldBufferDataFn(target, data, hint);
};
}(gl.bufferData.bind(gl)));
Now you can get the data with
data = buffers[someBuffer.id];
This is probably what the WebGL Inspector does
Note that there are a few issues with the code above. One it doesn't check for errors. Checking for errors would make it way slower but not checking for error will give you incorrect results if your code generates errors. A simple example
gl.bufferData(someBuffer, someData, 123456);
This would generate an INVALID_ENUM error and not update the data in someBuffer but our code isn't checking for errors so it would have put someData into buffers and if you read that data it wouldn't match what's in WebGL.
Note the code above is pseudo code. For example I didn't supply a wrapper for gl.bufferSubData.
WebGL2
In WebGL2 there is a function gl.getBufferSubData that will allow you to read the contents of a buffer. Note that WebGL2, even though it's based on OpenGL ES 3.0 does not support gl.mapBuffer because there is no performant and safe way to expose that function.

HTML5 FileWriter InvalidStateError

I'm using a canvas element to tile a number of video elements together in real time, which works fine. I'm also trying to grab the canvas data and write it to a file periodically, so I can encode a video from the file of raw frames when the video elements have finished playing.
I've got a function called on a timer interval every 40ms (giving 25fps) that roughly looks as follows (there's too much code to paste in its entirety):
function saveFrame() {
// code to copy from video elements to my canvas...
// ...
// Get the canvas image data and write it to a file
var imgData = canvasContext.getImageData(0, 0,
canvasElement.width,
canvasElement.height);
var b = new Blob([imgData], {type: 'application/octet-binary'});
// Write the blob to a file I previously opened
// fileWriter is a FileWriter that I obtained and saved previously
fileWriter.write(b);
}
setInterval(function() {
saveFrame();
}, 40);
Every time I hit the fileWriter.write(blob) statement I get the following error:
Uncaught InvalidStateError: An operation that depends on state cached
in an interface object was made but the state had changed since it was
read from disk.
Is this a timing issue? Can't the file writer API support writing every 40ms?
From docs of method write:
If readyState is WRITING, throw an InvalidStateError and terminate
this series of steps.
So you get InvalidStateError error because your fileWriter still write previous data.
Try to use writeend event and setTimeout:
function saveFrame() {
var imgData = canvasContext.getImageData(0, 0,
canvasElement.width,
canvasElement.height);
var b = new Blob([imgData], {
type: 'application/octet-binary'
});
fileWriter.write(b);
}
fileWriter.writeend = function () {
setTimeout(saveFrame, 40)
}
saveFrame();

Writing binary data html5 FileWriter

I am using webgl in an application and render something to an offscreen render target (frame buffer) and then do a readPixels. After getting the pixel data I convert it to jpeg using javascript jpeg encoder available here:
Javascript Jpeg Encoder
I get raw binary jpeg data that I want to write to a local file system. Here is the code that I am using for writing:
root.getFile(filename, {create: true}, function(fileEntry)
{
fileEntry.createWriter(function(writer)
{
writer.onwriteend = function(e)
{
System.debug("Write done");
};
writer.onerror = function(e)
{
System.error(e);
}
var data = new Blob(jpegData, { type: "image/jpeg" });
writer.write(data);
jpegData = null;
pixels = null;
uoozo.core.write[filename] = false;
});
},
function(e)
{
System.error(e);
});
However, I get weird results. jpegData is fine when I put it to an image element however when I create the blob e.g. jpegData is 3200 bytes in size but the Blob created is always 4818 bytes and FileWriter also returns the same position after the write operation. The jpeg created is obviously wrong and doesn't open. I don't understand how I can get the FileWriter to just write the darn binary data into the file without trying to be clever (or stupid).
Can someone please help me in this? Thank you.
So even after converting to Uint8Array I still wasn't getting the result that I needed. After looking at the dataURLToBlob method it dawned on me that I was passing the array object to the Blob constructor like:
var blob = new Blob(jpegData_ConvertTo_Uint8Array, "image/jpeg");
This isn't correct and apparently in order to get the Blob to work propery I need to pass an array like:
var blob = new Blob([jpegData_ConvertTo_Uint8Array], "image/jpeg");
This worked perfectly and write my jpeg to the file with a subsequent call to the FileWriter. write(blob) function. Thanks dandavis!

Resources