Browse Source

Initial commit

master
ar 3 years ago
commit
c327a7f6d3
12 changed files with 1203 additions and 0 deletions
  1. 8
    0
      .gitignore
  2. 5
    0
      CHANGELOG.md
  3. 7
    0
      LICENSE
  4. 57
    0
      README.md
  5. 454
    0
      matrix-recorder.js
  6. 98
    0
      node-decrypt-attachment.js
  7. 24
    0
      package.json
  8. 341
    0
      recorder-to-html.js
  9. 50
    0
      templates/index.html
  10. 51
    0
      templates/style.css
  11. 95
    0
      templates/timeline.html
  12. 13
    0
      templates/welcome.html

+ 8
- 0
.gitignore View File

@@ -0,0 +1,8 @@
test/
test.js
node_modules/
*.sqlite
.DS_Store
.idea
*.log
*.swp

+ 5
- 0
CHANGELOG.md View File

@@ -0,0 +1,5 @@
# Changelog for Matrix Recorder

## Changes in v0.0.1 (2017-01-20)

- Initial release

+ 7
- 0
LICENSE View File

@@ -0,0 +1,7 @@
Copyright 2017 Hello Matrix Team

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

+ 57
- 0
README.md View File

@@ -0,0 +1,57 @@
# Matrix Recorder
While [Matrix](https://matrix.org/) homeservers store your message history so that you can retrieve them on all your devices, currently there is no easy way to download and backup this message history. This is especially relevant for end-to-end (E2E) encrypted rooms because their history cannot be decrypted anymore if you lose your encryption keys (which is as easy as logging out of your Riot client).

This is where *Matrix Recorder* jumps in. This tool (written in [Node.js](https://nodejs.org/en/)) retrieves all messages you receive or send just as a regular client would. It uses your existing Matrix account but registers as a new device with its own encryption keys, which means that it can decrypt messages sent to you after Matrix Recorder has been registered. You don’t need to keep it running all the time but can just start it from time to time - it will continue from where it left off when you start it the next time.

All timeline events retrieved that way will be stored in an sqlite database which you can just keep as a backup or use to create your own archive. Matrix Recorder comes with a small utility that extracts messages from this database into HTML files that can be viewed in your browser.


## Installation
To use Matrix Recorder, you need to have installed a recent version of [Node.js](https://nodejs.org/en/) (recommended is at least v6.9) including the `npm` package manager. For easy checkout from gitlab you should also have an installation of `git`. With these tools in place, installing Matrix Recorder should be as easy as:

git clone https://gitlab.com/argit/matrix-recorder.git
cd matrix-recorder/
npm install

The last line will install all required dependencies in the `node_modules` subfolder.


## Usage
If Matrix Recorder is installed properly, you can use it by telling it where you want to store your session data and the retrieved message data (this should be an empty or to-be-created directory), for example like this (you need to run it from the folder you installed it in above):

node matrix-recorder.js /home/you/Chat_Archive

If you run it for the first time with this folder it will ask you for your home server URL (`https://matrix.org/` if you are using the official home server of the Matrix team), your username and password. It will also ask you whether it should go back in time and how many old messages it should retrieve for you - note, though, that it won’t be able to decrypt any old messages with active end-to-end encryption (E2E) as those messages would not have been encrypted for the new keys Matrix Recorder generates.

Matrix Recorder will then login to your homeserver, register as a new device (with a new set of keys), download the number of old messages you specified and continue listening. From then on, any new E2E messages sent to you will by default be encrypted also for your Matrix Recorder keys, so that it can decrypt and store these messages as well.

*Warning:* Yes, Matrix Recorder will decrypt your E2E messages, just like your Matrix client (e.g. Riot) will. You should thus only run Matrix Recorder on a trusted computer (ideally your own). There is no need to keep it running all the time (continue reading below), so it is neither necessary nor advisable to run it on some far away server or cloud machine!

Matrix Recorder will keep listening for new messages for as long as you keep it running. You can always quit it by pressing `Ctrl-C`. This does not mean that it will miss any messages, though - when you restart it with the same target directory it will just continue where it left off and retrieve all new messages that have arrived in the meantime. You do thus not have to keep it running continuously but can just start it from time to time to catch up with your new communications.

Matrix Recorder stores its access token (which it uses to access your home server on your behalf) and its encryption keys in plaintext in the `localstorage/` subfolder of the target directory you have specified. It will also record all messages received in plaintext in the sqlite database `messages.sqlite` in the target directory (after all, that’s the whole point of this tool!). Incoming media (images, video, files) will be decrypted (if necessary) and stored in the `media/` subdirectory. Thus, you might want to consider pointing Matrix Recorder to a directory on an encrypted volume to keep your private data secure.


## Viewing your messages
You can either directly access the `messages.sqlite` database to retrieve your messages, or you can use our `recorder-to-html.js` helper by calling it with the same target directory you have pointed Matrix Recorder at:

node recorder-to-html.js /home/you/Chat_Archive

This tool will create a new subfolder `html/` in the target directory. It will then go through the database and extract all messages into HTML files, which it will put in there. After the tool has completed, you can open the `html/index.html` file in a browser of your choice and you should be able to browse the stored room history. Feel free to modify the `style.css` stylesheet to customise the look to your taste (and if it looks great please send us a pull request!).

If new messages have come in and you want to update the HTML view you first need to remove the `html/` subfolder (because we do not want to overwrite anything by accident) and afterwards you can just call the above command again.


## Current Limitations and Contributions
- It’s alpha code - don’t use it for important archiving tasks.
- It doesn’t support any other homeserver authentication method than username and password.
- It cannot decrypt E2E messages that have been sent prior to your first run of Matrix Recorder (this is by design as obviously those messages won’t have been encrypted for Matrix Recorder, as an alternative it would have to be able to import your private keys e.g. from your Riot client, which is not supported yet).
- The `recorder-to-html.js` script currently will only process a very basic subset of message types - join, leave and invite membership events plus regular text messages, images, videos and generic file uploads.

Feel free to enhance the code, we are looking forward to any pull requests for Matrix Recorder.


## Questions?
If you have any questions, feel free to join [\#hello-matrix:matrix.org](https://matrix.to/#/#hello-matrix:matrix.org) for answers. If any questions come up frequently, we will add them here.



+ 454
- 0
matrix-recorder.js View File

@@ -0,0 +1,454 @@
"use strict";

// Debug?
const DEBUG = false;

// HELPER FUNCTIONS

// Ask questions to the user
// Promisified from https://st-on-it.blogspot.ca/2011/05/how-to-read-user-input-with-nodejs.html
var ask = function(question, format) {
var stdin = process.stdin, stdout = process.stdout;

return new Promise(function(resolve) {
stdin.resume();
stdout.write(question + ": ");

stdin.once('data', function (data) {
data = data.toString().trim();

if (format.test(data)) {
resolve(data);
} else {
stdout.write("It should match: " + format + "\n");
return ask(question, format);
}
});
});
};


// Runs an sqlite3 database query and returns a promise that provides the number of changes
var dbRunPromise = function(query) {
return new Promise(function(resolve, reject) {
db.run(query, function(err) {
if(err) {
reject(err);
} else {
resolve(this.changes);
}
});
});
};


// Creates a new directory if it does not yet exist
var mkdirIfNotExists = function(directory) {
try {
fs.mkdirSync(directory);
} catch(err) {
// Do nothing if directory already exists, exit otherwise
if(err.code !== 'EEXIST') {
console.log('Could not create directory %s: %s', directory, err.code);
process.exit(1);
}
}
};


// Decrypt a media file and store it on disk
// Adapted from decryptFile function in https://github.com/matrix-org/matrix-react-sdk/blob/57c56992f1e9443db2400c589cccdee877907fd8/src/utils/DecryptFile.js
var decryptFile = function(client, file) {
const url = client.mxcUrlToHttp(file.url);

// Download the encrypted file as an array buffer.
return fetch(url)
.then(function(response) {
return response.buffer();
}).then(function(responseData) {
// Decrypt the array buffer using the information taken from
// the event content.
return encrypt.decryptAttachment(responseData, file);
});
};

// Function to store file on filesystem (decrypt if necessary) and save its reference in the database
var storeFile = function(client, room_id, event_id, file, storageLocation) {
// Is file encrypted?
var storePromise;

if(file.key) {
console.log('Decrypting file %s...', file.url);
storePromise = decryptFile(client, file, storageLocation);
} else {
console.log('Storing unencrypted file %s...', file.url);
var url = client.mxcUrlToHttp(file.url);
storePromise = fetch(url)
.then(function(response) {
return response.buffer();
});
}

return storePromise
.then(function(responseData) {
// We write the decrypted / obtained file to disk
return new Promise(function (resolve, reject) {
var urlToPath = file.url.replace('mxc://', '') + '.' + (mime.extension(file.mimetype) || 'bin');
var fullPath = path.join(storageLocation, urlToPath);

// Do we need to create the subdirectory?
mkdirIfNotExists(path.dirname(fullPath));

// Let's write the file.
fs.writeFile(fullPath, responseData, function (err) {
if (err) {
reject(err);
} else {
resolve(urlToPath);
}
});
});
})
.then(
function(path) {
db.run('INSERT INTO files_stored (room_id, event_id, mxc_url, mimetype, filename) VALUES (?, ?, ?, ?, ?)',
room_id, event_id, file.url, file.mimetype, path,
function(err) {
if(err) {
console.log('SQLite error: %s', err);
console.log('Exiting...');
process.exit(1);
}
});
},
function(err) {
console.log('ERROR decrypting / storing file: %s', err);
console.log('Exiting...');
process.exit(1);
}
);
};



// Main function to process incoming room timeline events
var processRoomTimeline = function(event, room) {

// Recalculate room to be safe
room.recalculate(client.credentials.userId);

// Obtain content
var content = event.getContent();

// Debug output
if(DEBUG) {
// Log to console
console.log(
// the room name will update with m.room.name events automatically
"--------- (%s) %s ---------", room.name, event.getSender()
);

console.log('Room: %s (%s)', room.name, room.roomId);
console.log('Event ID: %s', event.getId());
console.log('Event Date: %s', event.getDate());
console.log('Event Sender: %s', event.getSender());
console.log('Event Sender Key: %s', event.getSenderKey());
console.log('Event Target: %s', (event.target ? event.target.userId : ''));
console.log('Event Type: %s', event.getType());
console.log('Encryption? %s', event.isEncrypted());

console.log('Content:');
console.log(JSON.stringify(content));

console.log('Unsigned event data:' + JSON.stringify(event.getUnsigned()));
}

// Output without debug data
console.log('[%s] %s: %s' + (event.isEncrypted() ? ' (encrypted)' : '') + ' received from %s for room "%s" (%s)', event.getDate(), event.getId(), event.getType(), event.getSender(), room.name, room.roomId);

// Are there any files that need to be retrieve?
if(content.file && content.file.url && content.file.url.substr(0, 6) === 'mxc://') {
// Encrypted files are being provided by matrix-js-sdk in the 'file' object (including key data for decrypt)
storeFile(client, room.roomId, event.getId(), content.file, mediaPath);
}

if(content.info && content.info.thumbnail_file && content.info.thumbnail_file.url && content.info.thumbnail_file.url.substr(0, 6) === 'mxc://') {
// Encrypted thumbnails are being provided by matrix-js-sdk in the 'thumbnail_file' object (including key data for decrypt)
storeFile(client, room.roomId, event.getId(), content.info.thumbnail_file, mediaPath);
}

if(content.url && content.info.mimetype && content.url.substr(0, 6) === 'mxc://') {
// For messages that are not encrypted, file data is provided via "url" property and "info.mimetype" property
// We need to construct the relevant parts of the "file" object ourselves
storeFile(client, room.roomId, event.getId(), { url: content.url, mimetype: content.info.mimetype }, mediaPath);
}

if(content.info && content.info.thumbnail_info && content.info.thumbnail_info.mimetype && content.info.thumbnail_url && content.info.thumbnail_url.substr(0, 6) === 'mxc://') {
// For messages that are not encrypted, thumbnail data is provided via "info.thumbnail_url" property and "info.thumbnail_info.mimetype" property
// We need to construct the relevant parts of the "file" object ourselves
storeFile(client, room.roomId, event.getId(), { url: content.info.thumbnail_url, mimetype: content.info.thumbnail_info.mimetype }, mediaPath);
}


// Write to SQLite database
db.run('INSERT INTO events_received (room_id, room_name, event_id, event_date, sender, sender_key, target, event_type, encrypted, content, unsigned_data) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
room.roomId,
room.name,
event.getId(),
event.getDate(),
event.getSender(),
event.getSenderKey(),
(event.target ? event.target.userId : ''),
event.getType(),
event.isEncrypted(),
JSON.stringify(content),
JSON.stringify(event.getUnsigned()),
function(err) {
if(err) {
console.log('SQLite error: %s', err);
console.log('Exiting...');
process.exit(1);
}
}
);

};





// INITIALIZATION STARTS HERE

// Use path and fs
const path = require('path');
const fs = require('fs');

// Use fetch
const fetch = require('node-fetch');

// Use node-encrypt-attachment, our adaptation of browser-encrypt-attachment
const encrypt = require('./node-decrypt-attachment');

// Use mime
const mime = require('mime-types');

// Has a directory been given on the command line?
// Otherwise show information message.
var targetDir = process.argv[2];

if(targetDir === undefined) {
console.log('You need to give Matrix Recorder the name of a directory it should use to store your user credentials (incl. E2E keys) and the recorded Matrix messages, e.g. like that:\n\n node matrix-recorder.js ./my_matrix_log\n\n');
process.exit(1);
}

// Path for storing any decrypted files
var mediaPath = path.join(targetDir, 'media');

// Do we need to create the path?
mkdirIfNotExists(targetDir);
mkdirIfNotExists(mediaPath);


// Loading localStorage module
if (typeof localStorage === "undefined" || localStorage === null) {
var LocalStorage = require('node-localstorage').LocalStorage;
var localStorage = new LocalStorage(path.join(targetDir, 'localstorage'));
}

// Load sqlite3 module
var sqlite3 = require('sqlite3').verbose();
var db = new sqlite3.Database(path.join(targetDir, 'messages.sqlite'));


// Items to retrieve for initial sync (we overwrite for initial setup, otherwise get zero items to avoid missing out on duplicates).
var is_initial_run = false;
var items_to_retrieve = 0;


// Our initialization promise
var initPromise;
var authData = {};


// Do we already have data in the localStorage? Then we can start the Matrix SDK right away.
if(localStorage.getItem('baseUrl') && localStorage.getItem('accessToken')) {
// Yes, let's go.
initPromise = Promise.resolve();

} else {
// No, we need to populate this data from user input.
is_initial_run = true;
initPromise =
ask('Your homeserver (give full URL)', /^https?:\/\/[a-z0-9A-Z\.-]+\/.*$/).then(
function(homeserver) {
localStorage.setItem('baseUrl', homeserver.replace(/\/$/, ""));

// Ask for username
return ask('Your username at the homeserver', /^.+$/);
}
).then(
function(user) {
authData.user = user;

// Ask for password
return ask('Your password at the homeserver', /^.+$/);
})
.then(
function(password) {
authData.password = password;

// Ok, let's start login process to obtain accessToken and deviceId.
// We use a new temporary matrix client for this
authData.client = sdk.createClient({
baseUrl: localStorage.getItem('baseUrl')
});

console.log('Trying to log in...');
return authData.client.login('m.login.password', {user: authData.user, password: authData.password, initial_device_display_name: 'Matrix Recorder' });
}
)
.then(
function(res) {
// Console log
console.log('Logged in as ' + res.user_id);
localStorage.setItem('userId', res.user_id);
localStorage.setItem('accessToken', res.access_token);
localStorage.setItem('deviceId', res.device_id);

// Ok, ask for number of items to retrieve for initial download.
return ask('No of items to retrieve for initial sync', /^([0-9]*)$/);
},
function(err) {
console.log('Error when trying to log in: ' + JSON.stringify(err));
process.exit(1);
}
)
.then(
function(_items_to_retrieve) {
// Default to 100.
items_to_retrieve = Number(_items_to_retrieve);
if(isNaN(items_to_retrieve)) { items_to_retrieve = 100; }
}
).then(
function() {
// Okay, we are done with the questions. We need to initialize the SQlite database, though, if it does not yet
// contain the relevant tables.

return new Promise(function (resolve, reject) {
db.all('SELECT name FROM sqlite_master WHERE type="table"', function (err, rows) {
if(err) {
reject(err);
} else {
var existingTables = rows.map(function(d) { return d['name']; });

var createTables = [];

// Create events_received table if it does not yet exist
if(existingTables.indexOf('events_received') === -1) {
createTables.push(dbRunPromise("CREATE TABLE events_received (room_id VARCHAR(300), room_name VARCHAR(500), event_id VARCHAR(300), event_date DATETIME, sender VARCHAR(300), sender_key VARCHAR(200), target VARCHAR(300), event_type VARCHAR(200), encrypted BOOLEAN, content BLOB, unsigned_data BLOB)"));
}

// Create files_stored table if it does not yet exist
if(existingTables.indexOf('files_stored') === -1) {
createTables.push(dbRunPromise("CREATE TABLE files_stored ( room_id VARCHAR(300), event_id VARCHAR(300), mxc_url VARCHAR(300), mimetype VARCHAR(100), filename VARCHAR(300) )"));
}

resolve(Promise.all(createTables));
}
});
});

}
);
}


// Loading Matrix SDK
var sdk = require("matrix-js-sdk");
var client = undefined;


initPromise.then(function() {
// Create in memory store
var matrixStore = new sdk.MatrixInMemoryStore();

// Create client
client = sdk.createClient({
baseUrl: localStorage.getItem('baseUrl'),
accessToken: localStorage.getItem('accessToken'),
userId: localStorage.getItem('userId'),
sessionStore: new sdk.WebStorageSessionStore(localStorage),
store: matrixStore,
deviceId: localStorage.getItem('deviceId')
});

console.log('Starting client for initial sync...');
client.startClient({ initialSyncLimit: items_to_retrieve });

if(is_initial_run) {
// For the initial run, we will also capture events from the initial sync (backfill)
// Afterwards, we will ignore those
client.on('Room.timeline', processRoomTimeline);
}

client.on('sync', function(state, prevState, data) {
if(state === 'PREPARED') {
// This state is emitted when the first initial sync completes
// We immediately stop the client.
console.log('Stopping client after initial sync...');
client.stopClient();

} else if(state === 'STOPPED') {
// Ok, the client is stopped. We remove this listener and restart the client with
// proper sync token and large initialSyncLimit
client.removeAllListeners('sync');

// We set the syncToken where we want to start.
// If we do not have a sync token, we will start from the prior initial sync
// (we will never get old stuff at the moment).
if(localStorage.getItem('syncToken')) {
matrixStore.setSyncToken(localStorage.getItem('syncToken'));
} else {
localStorage.setItem('syncToken', client.store.getSyncToken());
}

console.log('Client stopped, re-starting with right sync token and large initialSyncLimit...');

// Retrieve all timeline messages and store them in the SQlite database (if listener is not yet set)
if(!is_initial_run) {
client.on('Room.timeline', processRoomTimeline);
}

client.on('sync', function(state, prevState, data) {
if(state === 'SYNCING') {
// Update sync token in storage
console.log('Updating sync token in storage to %s', client.store.getSyncToken());
localStorage.setItem('syncToken', client.store.getSyncToken());
} else if(state === 'ERROR') {
console.log('ERROR while trying to sync messages: %s', data.error);
console.log('Exiting...');
process.exit(1);
}
});

// Let's start the client.
client.startClient({ initialSyncLimit: 100000 });


} else if(state === 'ERROR') {
console.log('ERROR while trying to sync messages: %s', data.error);
console.log('Exiting...');
process.exit(1);
}
});









});

+ 98
- 0
node-decrypt-attachment.js View File

@@ -0,0 +1,98 @@
/****
ADAPTED FROM https://github.com/matrix-org/browser-encrypt-attachment
The "browser-encrypt-attachment" requires the WebCrypto API, which is not available on node.
This file recreates the decryptAttachment function using the node crypto library.
****/


// Crypto from Node
var crypto = require('crypto');


// Fill-in for atob
// From: https://github.com/node-browser-compat/atob

function atob(str) {
return new Buffer(str, 'base64').toString('binary');
}


/**
* Decrypt an attachment.
* @param {ArrayBuffer} ciphertextBuffer The encrypted attachment data buffer.
* @param {Object} info The information needed to decrypt the attachment.
* @param {Object} info.key AES-CTR JWK key object.
* @param {string} info.iv Base64 encoded 16 byte AES-CTR IV.
* @param {string} info.hashes.sha256 Base64 encoded SHA-256 hash of the ciphertext.
* @return {Promise} A promise that resolves with an ArrayBuffer when the attachment is decrypted.
*/
function decryptAttachment(ciphertextBuffer, info) {

if (info === undefined || info.key === undefined || info.iv === undefined
|| info.hashes === undefined || info.hashes.sha256 === undefined) {
throw new Error("Invalid info. Missing info.key, info.iv or info.hashes.sha256 key");
}

var ivBuffer = new Buffer(decodeBase64(info.iv));
var keyBuffer = new Buffer(decodeBase64(info.key.k));
var expectedSha256base64 = info.hashes.sha256;

// Is the SHA256 correct?
var hash = crypto.createHash('sha256');
hash.update(ciphertextBuffer);

var resultingHash = hash.digest('base64');
if(!expectedSha256base64 || resultingHash.substr(0, expectedSha256base64.length) != expectedSha256base64) {
// We substr the resultingHash, as it is given in base64 with padding while the expected one is
// without padding.
throw new Error("Mismatched SHA-256 digest");
}

/* TODO - This is from matrix-org/browser-encrypt-attachment, but I really do not know what to do with it
var counterLength;
if (info.v == "v1" || info.v == "v2") {
// Version 1 and 2 use a 64 bit counter.
counterLength = 64;
} else {
// Version 0 uses a 128 bit counter.
counterLength = 128;
}
*/

// Use node crypto library - Decipher object
var decipher = crypto.createDecipheriv('aes-256-ctr', keyBuffer, ivBuffer);

return Promise.resolve(Buffer.concat([decipher.update(ciphertextBuffer) , decipher.final()]));
}


/**
* Decode a base64 string to a typed array of uint8.
* This will decode unpadded base64, but will also accept base64 with padding.
* @param {string} base64 The unpadded base64 to decode.
* @return {Uint8Array} The decoded data.
*/
function decodeBase64(base64) {
// Pad the base64 up to the next multiple of 4.
var paddedBase64 = base64 + "===".slice(0, (4 - base64.length % 4) % 4);
// Decode the base64 as a misinterpreted Latin-1 string.
// atob returns a unicode string with codepoints in the range 0-255.
var latin1String = atob(paddedBase64);
// Encode the string as a Uint8Array as Latin-1.
var uint8Array = new Uint8Array(latin1String.length);
for (var i = 0; i < latin1String.length; i++) {
uint8Array[i] = latin1String.charCodeAt(i);
}
return uint8Array;
}

try {
exports.decryptAttachment = decryptAttachment;
}
catch (e) {
// Ignore unknown variable "exports" errors when this is loaded directly into a browser
// This means that we can test it without having to use browserify.
// The intention is that the library is used using browserify.
}


+ 24
- 0
package.json View File

@@ -0,0 +1,24 @@
{
"name": "matrix-recorder",
"version": "0.0.1",
"description": "A recorder that can record Matrix rooms you are a member of (including E2E-encrypted rooms).",
"author": "Hello Matrix <[email protected]>",
"main": "matrix-recorder.js",
"scripts": {
"start": "node matrix-recorder.js"
},
"repository": {
"type": "git",
"url": "https://gitlab.com/argit/matrix-recorder.git"
},
"dependencies": {
"marked": "^0.3.6",
"matrix-js-sdk": "^0.7.4",
"mime-types": "^2.1.14",
"mustache": "^2.3.0",
"node-fetch": "^1.6.3",
"node-localstorage": "^1.3.0",
"sqlite3": "^3.1.8"
},
"license": "MIT"
}

+ 341
- 0
recorder-to-html.js View File

@@ -0,0 +1,341 @@
"use strict";


//// INIT STARTS HERE

// Use path and fs
const path = require('path');
const fs = require('fs');

// Use marked
const marked = require('marked');

// Has a directory been given on the command line?
// Otherwise show information message.
var targetDir = process.argv[2];

if(targetDir === undefined) {
console.log('You need to give Recorder-to-HTML the name of the Matrix Recorder directory that contains the sqlite database to convert, like that:\n\n node recorder-to-html.js ./my_matrix_log\n\n');
process.exit(1);
}

// Does target directory exist?
try {
fs.accessSync(targetDir, fs.constants.R_OK | fs.constants.W_OK);
} catch(err) {
console.log('The directory you have specified does not exist or you do not have read and write access there.');
console.log(err);
process.exit(1);
}

// Does target sqlite database exist?
try {
fs.accessSync(path.join(targetDir, 'messages.sqlite'), fs.constants.R_OK | fs.constants.W_OK);
} catch(err) {
console.log('The directory you have specified does not exist or you do not have read and write access there.');
console.log(err);
process.exit(1);
}

// Does html sub-directory *not* exist?
// Otherwise we fail as we do not want to accidentally overwrite anything.
// Create html directory if it does not yet exist.
try {
fs.mkdirSync(path.join(targetDir, 'html'));
} catch(err) {
if(err.code === 'EEXIST') {
console.log('The html/ directory already exists. We will stop now because we do not want to accidentally overwrite anything. Please remove html/ manually first if you want to re-create it.');
} else {
console.log('Could not create html directory: %s', directory, err.code);
}

process.exit(1);
}



// Load sqlite3 module
const sqlite3 = require('sqlite3').verbose();
var db = new sqlite3.Database(path.join(targetDir, 'messages.sqlite'));


// Load mustache
const Mustache = require('mustache');


// Global variables
var roomNames = {};
var roomFiles = {};
var rooms = [];
var fileNames = {};

// Copy static files style.css, welcome.html
console.log('Copying style.css...');
copyFile('templates/style.css', path.join(targetDir, 'html', 'style.css'))
.then(
function() {
console.log('Copying welcome.html...');
return copyFile('templates/welcome.html', path.join(targetDir, 'html', 'welcome.html'));
}
)

// Process files in the database
.then(
function() {
console.log('Retrieving all files specified in the database...');
return promiseDbAll("SELECT DISTINCT mxc_url, filename FROM files_stored WHERE mxc_url IS NOT NULL AND filename IS NOT NULL ORDER BY mxc_url");
}
)
.then(
function(rows) {
// Ok, we have received all files.
console.log('Received files. Processing.');

rows.forEach(function(row) {
fileNames[row['mxc_url']] = '../media/' + row['filename'];
});

return rows.length;
}
)

// Process rooms from database
.then(
function() {
// Note that we only show rooms with at least one message in them.
console.log('Retrieving all rooms from database...');
return promiseDbAll("SELECT DISTINCT room_id, room_name, event_date FROM events_received WHERE event_type='m.room.message' ORDER BY room_id, event_date DESC");
}
)
.then(
function(rows) {
// Ok, we have received room data.
console.log('We have received room data. Processing:');

var prevRoom = '';
rows.forEach(function(row) {
if(prevRoom !== row['room_id']) {
// We only look at the first line for each room_id, which will be the newest room name
roomNames[row['room_id']] = row['room_name'];
roomFiles[row['room_id']] = row['room_id'].replace(/[^a-zA-Z0-9\.-]/g, '_') + '.html';

rooms.push({
room_id: row['room_id'],
room_name: row['room_name'],
room_file: roomFiles[row['room_id']]
});

prevRoom = row['room_id'];
}
});

// Ok, now write the room file using Mustache...
mustacheRender('templates/index.html', path.join(targetDir, 'html', 'index.html'), { rooms: rooms });

// Ok, now retrieve all timelines events for all rooms...
return new Promise(function(resolve, reject) {
var prevRoom = '';
var roomTimeline = [];
db.each("SELECT * FROM events_received WHERE event_type IN ('m.room.member', 'm.room.message') ORDER BY room_id, event_date",
// Called on every row
function(err, row) {
if(err) {
reject(err);
} else {
// Skip rooms which don't have any messages in them (and for which we thus have no file name)
if(!roomFiles[row['room_id']]) {
return;
}

// Render previous room
if(prevRoom !== row['room_id']) {
if(prevRoom) {
renderRoom(prevRoom, roomNames[prevRoom], path.join(targetDir, 'html', roomFiles[prevRoom]), roomTimeline);
}

prevRoom = row['room_id'];
roomTimeline = [];
}

// Ok, add data to timeline.
if(row['event_type'] === 'm.room.member') {
var eventDetails = {
type: row['event_type'],
date: new Date(row['event_date']),
sender: row['sender'],
targetFallbackSender: (row['target'] ? row['target'] : row['sender']),
data: JSON.parse(row['unsigned_data'])
};

eventDetails['membership_' + JSON.parse(row['content']).membership] = true;
roomTimeline.push(eventDetails);

} else if(row['event_type'] === 'm.room.message') {
var eventDetails = {
type: row['event_type'],
date: new Date(row['event_date']),
sender: row['sender'],
targetFallbackSender: (row['target'] ? row['target'] : row['sender']),
data: JSON.parse(row['content'])
};

if(eventDetails.data.msgtype === 'm.text') {
// Regular text
eventDetails['message'] = true;

} else if(eventDetails.data.msgtype === 'm.image') {
// Image
eventDetails['message_image'] = true;

} else if(eventDetails.data.msgtype === 'm.video') {
// Video
eventDetails['message_video'] = true;

} else if(eventDetails.data.msgtype === 'm.file') {
// File
eventDetails['message_file'] = true;

}

// File?
if(eventDetails.data.file && eventDetails.data.file.url && fileNames[eventDetails.data.file.url]) {
// Encrypted files are being provided by matrix-js-sdk in the 'file' object (including key data for decrypt)
eventDetails['file'] = fileNames[eventDetails.data.file.url];
} else if(eventDetails.data.url && fileNames[eventDetails.data.url]) {
// For messages that are not encrypted, file data is provided via "url" property
eventDetails['file'] = fileNames[eventDetails.data.url];
}

// Thumbnail?
if(eventDetails.data.info && eventDetails.data.info.thumbnail_file && eventDetails.data.info.thumbnail_file.url && fileNames[eventDetails.data.info.thumbnail_file.url]) {
// Encrypted thumbnails are being provided by matrix-js-sdk in the 'thumbnail_file' object
eventDetails['thumbnail_file'] = fileNames[eventDetails.data.info.thumbnail_file.url];
} else if(eventDetails.data.info && eventDetails.data.info.thumbnail_info && eventDetails.data.info.thumbnail_url && fileNames[eventDetails.data.info.thumbnail_url]) {
// For messages that are not encrypted, thumbnail data is provided via "info.thumbnail_url" property
eventDetails['thumbnail_file'] = fileNames[eventDetails.data.info.thumbnail_url];
}

roomTimeline.push(eventDetails);

}

// Done.
}

// Done processing row.
},
// Called on completion
function(err, numRows) {
if(err) {
reject(err);
} else {
// Render last room
renderRoom(prevRoom, roomNames[prevRoom], path.join(targetDir, 'html', roomFiles[prevRoom]), roomTimeline);
resolve(numRows);
}
}
);
});
}
)
.then(
function() {
// Done. Show filename.
console.log('Done.');
console.log('The formatted timeline is available at: %s', path.join(targetDir, 'html', 'index.html'));
},
function(err) {
// An error occured somehwere
console.log('An ERROR occured: %s', err);
process.exit(1);
}
);








//// HELPER FUNCTIONS


// File copy
// Adapted from: https://stackoverflow.com/questions/11293857/fastest-way-to-copy-file-in-node-js

function copyFile(source, target) {
return new Promise(function(resolve, reject) {
var rd = fs.createReadStream(source);

rd.on("error", function(err) {
reject(err);
});
var wr = fs.createWriteStream(target);
wr.on("error", function(err) {
reject(err);
});
wr.on("close", function(ex) {
resolve();
});

rd.pipe(wr);
});
}


// Promised version of db.all
function promiseDbAll(query) {
return new Promise(function(resolve, reject) {
db.all(query, function (err, rows) {
if (err) {
reject(err);
} else {
resolve(rows);
}
});
});
}


// Promised function for Mustache that opens template, applies mustache and writes to output file
function mustacheRender(input, output, data) {
return new Promise(function(resolve, reject) {
fs.readFile(input, 'utf-8', function(err, inputHtml) {
if(err) {
reject(err);
} else {
var outputHtml = Mustache.render(inputHtml, data);
fs.writeFile(output, outputHtml, function(err) {
if(err) {
reject(err);
} else {
resolve(output);
}
});
}
});
});
}


function renderRoom(room_id, room_name, room_file, timeline) {
if(room_id !== '' && room_file) {
return mustacheRender('templates/timeline.html', path.join(targetDir, 'html', roomFiles[room_id]),
{
room_id: room_id,
room_name: room_name,
room_file: room_file,
timeline: timeline,

dateFormat: function() {
return this.date.toISOString().replace('T', ' ').replace(/\.[0-9]+Z$/, '');
},

bodyFormat: function() {
return marked(this.data.body, { sanitize: true });
}
}
);
}
}

+ 50
- 0
templates/index.html View File

@@ -0,0 +1,50 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Matrix Recorder - Recordings</title>

<style type="text/css">
body {
font-family: San Francisco, Arial, Helvetica, sans-serif;
font-size: small;
}

.room-list {
position: absolute;
top: 0;
left: 0;
width: 20%;
height: 98%;
background-color: #c2d5ff;
overflow-y: scroll;
}

.timeline iframe {
position: absolute;
top: 0;
left: 22%;
width: 76%;
height: 98%;
border: 0;
}

</style>

</head>

<body>
<div class="room-list">
<ul>
{{#rooms}}
<li><a href="{{room_file}}" target="timeline">{{room_name}}</a></li>
{{/rooms}}
</ul>
</div>

<div class="timeline">
<iframe src="welcome.html" name="timeline" id="timeline_iframe"></iframe>
</div>
</body>

</html>

+ 51
- 0
templates/style.css View File

@@ -0,0 +1,51 @@
body {
font-family: San Francisco, Arial, Helvetica, sans-serif;
font-size: small;
}

.timeline-main {
display: table;
border-collapse: collapse;
}

.timeline {
display: table-row;
border-bottom: 1px solid #ddd;
}

.timeline-date {
display: table-cell;
white-space: nowrap;
vertical-align: top;
padding-right: 10px;
padding-top: 5px;
padding-bottom: 5px;
}

.timeline-sender {
display: table-cell;
white-space: nowrap;
max-width: 200pt;
overflow: hidden;
text-overflow: ellipsis;
vertical-align: top;
padding-right: 10px;
padding-top: 5px;
padding-bottom: 5px;
}

.timeline-body {
display: table-cell;
vertical-align: top;
padding-top: 5px;
padding-bottom: 5px;
}

.timeline-body p {
margin: 0;
}

.membership-join, .membership-leave, .membership-invite {
font-style: italic;
color: #666666;
}

+ 95
- 0
templates/timeline.html View File

@@ -0,0 +1,95 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Matrix Recorder - {{room_name}} ({{room_id}})</title>
<link rel="stylesheet" type="text/css" href="style.css">
</head>

<body>
<h1>{{room_name}}</h1>

<div class="timeline-main">
{{#timeline}}

{{#membership_invite}}
<!-- invite event -->
<div class="timeline membership-invite">
<p><div class="timeline-date">[{{dateFormat}}]</div> <div class="timeline-sender mxid">{{sender}}</div> <div class="timeline-body">has invited <span class="mxid">{{targetFallbackSender}}</span>.</div></p>
</div>

{{/membership_invite}}


{{#membership_join}}
<!-- join event -->
<div class="timeline membership-join">
<p><div class="timeline-date">[{{dateFormat}}]</div> <div class="timeline-sender mxid">{{targetFallbackSender}}</div> <div class="timeline-body">has joined.</div></p>
</div>

{{/membership_join}}


{{#membership_leave}}
<!-- leave event -->
<div class="timeline membership-leave">
<p><div class="timeline-date">[{{dateFormat}}]</div> <div class="timeline-sender mxid">{{targetFallbackSender}}</div> <div class="timeline-body">has left or declined the invitation.</div></p>
</div>

{{/membership_leave}}


{{#message}}
<!-- generic message event -->
<div class="timeline message">
<p><div class="timeline-date">[{{dateFormat}}]</div> <div class="timeline-sender"><span class="mxid">{{sender}}</span></div> <div class="timeline-body">{{{bodyFormat}}}</div></p>
</div>

{{/message}}


{{#message_image}}
<!-- image received event -->
<div class="timeline message-image">
<p><div class="timeline-date">[{{dateFormat}}]</div> <div class="timeline-sender"><span class="mxid">{{sender}}</span></div>
<div class="timeline-body">
<a href="{{file}}"><img class="timeline-image" src="{{thumbnail_file}}"><br />
{{{bodyFormat}}}</a>
</div>
</p>
</div>

{{/message_image}}


{{#message_video}}
<!-- video received event -->
<div class="timeline message-video">
<p><div class="timeline-date">[{{dateFormat}}]</div> <div class="timeline-sender"><span class="mxid">{{sender}}</span></div>
<div class="timeline-body">
<video class="timeline-video" src="{{file}}" poster="{{thumbnail_file}}" controls></video><br />
{{{bodyFormat}}}</p>
</div>
</div>

{{/message_video}}


{{#message_file}}
<!-- file received event -->
<div class="timeline message-video">
<p><div class="timeline-date">[{{dateFormat}}]</div> <div class="timeline-sender"><span class="mxid">{{sender}}</span></div>
<div class="timeline-body">
<a href="{{file}}">{{{bodyFormat}}}</a>
</div></p>
</div>

{{/message_file}}

{{/timeline}}
</div>

<p>Internal room ID: {{room_id}}</p>
</body>

</html>

+ 13
- 0
templates/welcome.html View File

@@ -0,0 +1,13 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Matrix Recorder - Welcome</title>
<link rel="stylesheet" type="text/css" href="style.css">
</head>

<body>
<p>Welcome to your Matrix Recorder recordings. Please select a room on the left.</p>
</body>

</html>

Loading…
Cancel
Save