Encryption of messages

Tue, 12 Mar 2019 05:46:56 +0000

Do I understand correctly that there is no encryption of correspondence in Hubzilla?
All encryption is reduced to obfuscation algorithm rot47?
Why don't you use aes256 for encryption? If interested, I can share my thoughts on how to implement it.
Thank you.
//Thanks google translator.

show all 29 Comments

hEARt PhoniX

Tue, 12 Mar 2019 05:58:52 +0000 last edited: Tue, 12 Mar 2019 06:19:22 +0000

Once all non-public posts were stored encrypted in the DB. This was deprecated as ist slowed down the system quite a bit.
There were voices that would also like to see it back, me included. As a compromise it could be an optional addon./
@M. Dent

Mario Vavti

Tue, 12 Mar 2019 06:25:52 +0000

The problem was that the decryption keys had also to be saved in the db an it was possible to decrypt the data anyway with not too much effort. The drawbacks were as @h.ear.t mentioned performance and not being able to search private content.

The data is strong encrypted in transition though (server to server).

Eugene Lisovsky

Tue, 12 Mar 2019 07:14:05 +0000

The problem was that the decryption keys had also to be saved in the db an it was possible to decrypt the data anyway with not too much effort.

What if for each conversation to generate a key for encryption, which is stored in the database in encrypted form?
And the key is encrypted with a password. the sender encrypts the key with his password, and the recipient with his password.

Mario Vavti

Tue, 12 Mar 2019 08:39:12 +0000

the sender encrypts the key with his password, and the recipient with his password.

Do you mean something like that?

Eugene Lisovsky

Tue, 12 Mar 2019 09:35:54 +0000

Do you mean something like that?
Some encrypted content

Yes, but in my proposal all encryption will be transparent to the user.
Creating a key for correspondence occurs at the time of contact. And the key can have different versions, so the message should contain a link to the version.

Tue, 12 Mar 2019 18:18:35 +0000 last edited: Tue, 12 Mar 2019 18:20:17 +0000

I believe Eugene mostly means database encryption for private messages but not all other posts and publications (which probably doesn't make any sense as it concerns social networks).

Mario Vavti

Tue, 12 Mar 2019 18:23:45 +0000

But still, if i understand correctly the key is saved in the DB. Encrypting the key with the password would mean that for each pageload you would need to enter your password if i understand correctly.

Tue, 12 Mar 2019 18:30:14 +0000

I've been thinking about this for a while and have run through several versions of it in my head based on some reading in other messaging contexts. I haven't personally settled on a "right way" yet, but I do think I'm getting a bit closer. I've also reached out to a couple people for some help (including DJBerstein, one of my heros) - unforutunately - they are "big names" and have huge inboxes. I'm hopeful they will run across the messages and throw me a bone, but I'm not completely holding by breath.

The performance issues should be much reduced with the significant gains made by PHP. Personally, I'd like to move to more frequent message passing from the server to the client and perhaps even move message decryption to the client.

Fulltext searching is definitely a huge hurdle. There are some ways around it but again it comes down to performance tradeoffs - which on a fairly good sized database could be quite significant.

Unfortunately with the state of international affairs and various international laws restricting even conversation about these issues, those in the community who are experienced in this area are not necessarily at liberty to discuss them.

I think it'd be great if we could establish at least a small workgroup that has at least one or two people with a legitimate background to vet ideas and kick thoughts around with. Hopefully we could come up with some criteria to measure an acceptable solution by and then work through the various different thoughts and ideas to find a solution that fits.

Tue, 12 Mar 2019 18:33:41 +0000 last edited: Tue, 12 Mar 2019 18:37:29 +0000

If I understand Eugene idea correctly (we had a small discussion before in russian) for PM we can store shared key created for each conversation, encrypted by participant password and stored on each side. On login this key will be decrypted and temporary stored e.g. in sessions table so user will get ability to read encrypted messages.

Mario Vavti

Tue, 12 Mar 2019 18:42:19 +0000 last edited: Tue, 12 Mar 2019 18:44:06 +0000

On login this key will be decrypted and temporary stored e.g. in sessions table so user will get ability to read encrypted messages

So in most cases the key would be unencrypted in the DB for some hours a day?

IMHO if it's not safe it's not worth the hassle.

Tue, 12 Mar 2019 18:42:33 +0000

I'd like to move to more frequent message passing from the server to the client and perhaps even move message decryption to the client.

Tue, 12 Mar 2019 18:45:53 +0000 last edited: Tue, 12 Mar 2019 18:48:39 +0000

@Mario Vavti Yes, this is the same I thought first. As @M. Dent said above the right way will be to move decryption on client side. But is we will use different shared key for each conversation this will require to input password on every PM conversation open to decrypt it.

Mario Vavti

Tue, 12 Mar 2019 18:47:43 +0000 last edited: Tue, 12 Mar 2019 18:48:08 +0000

Yeah! Lets write an OMEMO plugin for the browser :smile_cat:

Tue, 12 Mar 2019 19:01:52 +0000

i[f] we will use different shared key for each conversation this will require to input password on every PM conversation open to decrypt it.

I don't think that's necessarily true. I'm thinking something along the lines of the browser based wallets of various crypto currencies. The Bitshares wallet, for instance, could be used as a roadmap, I think. Though it may of necessity be an addon for licensing reasons.

Yeah! Lets write an OMEMO plugin for the browser

Not familiar with it... however from a quick read sounds very similar to some of my thoughts.

I've got a backlog to dig through first though. :(

Tue, 12 Mar 2019 19:09:04 +0000

Creating a key for correspondence occurs at the time of contact. And the key can have different versions, so the message should contain a link to the version.

This wouldn't have been nomadically possible before (at least not natively through the ZOT protocol interface functions). But now that we have nomadic content addressing, it poses an interesting possibility.... At the very least the new nomadic content stuff gives us a roadmap for an implementation.

Hmmmmm.....

Eugene Lisovsky

Wed, 13 Mar 2019 01:53:14 +0000

I suggest to encrypt only private messages, not public activity in the channel.

But is we will use different shared key for each conversation this will require to input password on every PM conversation open to decrypt it.

not if we save the password in encrypted form in cookies in the browser. The string( session on serverside plus random string in cookie) can be the key for encryption. the server will be able to decrypt the password, they will decrypt the key for correspondence, and the key will decrypt the correspondence.

Wed, 13 Mar 2019 04:40:46 +0000

I was thinking a bit of a different take:

1) Sender client creates keypair for this specific message.
2) encrypt outgoing message using key-1
3) gather public keys of recipients (including self)
4) Put "key-2" in a structure and encrypt the structure using "recipient" public keys.
* key-2
* recipient_xchan
* some random junk so that each structure is truly unique
5) Include "key-2" structures in iconfig data (indexed in some way so that each recipient can easily find their own but without disclosing the other recipients) - OR - (use a variation of nomadic content addressing to allow the recipient to get their "key-2" structure from the sender server) - OR - (create some sort of "recipient-specific" extension to iconfig to avoid sending ALL the keys to ALL - only send a given structure to a given recipient)
6) Recipient server decrypts key-2 structure and sends key-2 along with the encryptied message to the recipient client ---OR--- Recipient server sends recipient private key to client to do all decryption client side
7) Recipient client decrypts message using key-2 before displaying it.

Items 3 & 4 could be done by the sender client or the sender hub with slight modifications.

My personal desire would be to move as much of the encryption/decryption to the client side as possible so the server is not bogged down - also maybe consider content caching in persistent storage on the client to avoid the need to decrypt the same message with every page reload.

I'm not sure how I feel about the recipient hub transmitting the recipient private key to the recipient client. Obviously it's all over https - and maybe it'd be possible to set up some sort of secondary session encryption between the recipient client and server for some added protection if desired, But I'm still leery of sending that data of the hub. (Of course, we're already doing so when we create clones, and it's already available in the channel table so maybe I'm trying too hard.)

Of course, even with the system outlined, it's POSSIBLE for a hubmin to decrypt the message - and all the decryption data needed to do so would be in the database of the hub (or accessible to it) but the complexity to avoid that becomes quite burdensome - and really - to what end? If you are THAT concerned about your hubmin reading your data and/or disclosing it to 3rd parties, you really need to host your own hub.

El Dorado

Thu, 14 Mar 2019 05:42:55 +0000

Encrypted storage - cryptmount

You'll also need this for the filesystem containing the database. You'll have shit performance and shit security if you try and do this in PHP.

Encrypted communications - GPG via API.

The base software is ready for a GPG API client and has been for a year or two. I created some tests to verify it would work (and it does), however (and this is important) you will need to create this client yourself. I live in Australia and there are a number of factors which make it impractical for me to take an active role in making this happen. From a UX viewpoint, any message/item which is created with a non-standard (undisplayable) MIMEtype is offered for download (manual decryption) or you can access these items via the existing stream API calls or zotfeed. Public keys for your connections can be stored in abconfig or xconfig and/or stored in your profile if you need to have them acccessible on the server in order to associate them with your connections.

An important factor is that you cannot (MUST NOT) display HTML content that has been end-to-end encrypted on a server based webpage without putting a lot of thought into it. It creates a perfect malware delivery system. If you know about security you will understand. The old JS E2EE feature has a basic bbcode converter to prevent it being used for malware delivery, although it is still possible. If you are really interested in security, the fact that this feature currently uses cryptoJS should be flashing red warning flags but it's a useful demo and that's what it was intended to be. I proved E2EE in a server based federated social network can be done. No more. No less. SJCL and rabbit hole encryption would be much better choices, but JS approaches have too many inherent flaws to put serious effort into as it's difficult but not impossible for state actors to intercept and replace the JS in transit on both ends of the connection. That's about as much as I can say on the subject without putting myself in serious legal trouble. There is code in Hubzilla for encrypting messages in transit (S2S). Everything else should be done on the client or the filesystem. You really do not want this code in Hubzilla and if somebody argues that this is where it should be I would question their motives.

Sophie

Thu, 14 Mar 2019 08:46:04 +0000

Yeah! Lets write an OMEMO plugin for the browser

Matrix also uses a version of the Signal protocol, (Megolm), which is compatible with group chats by default.
I still think it is very hard to pull that off in a "browser app" (not sure if that term even exists).

An AES-based cryptographic ratchet intended for group communications. The Megolm ratchet is intended for encrypted messaging applications where there may be a large number of recipients of each message, thus precluding the use of peer-to-peer encryption systems such as Olm. It also allows a recipient to decrypt received messages multiple times. For instance, in client/server applications, a copy of the ciphertext can be stored on the (untrusted) server, while the client need only store the session keys.

Thu, 14 Mar 2019 11:27:54 +0000

Can be useful
https://www.w3.org/News/2013.html#entry-9675

Thu, 14 Mar 2019 11:56:41 +0000

I guess when we talk encryption, the question has to be (like anything else) "what problem are you trying to solve?" Or, more specifically, what data are you trying to secure, from whom and under what circumstances? And of course, what is it "worth"? (How much are you able/willing to spend to protect it?)

Obviously a single solution that solves everything would be great! And a solid E2EE system that merely uses Hubzilla//Zot as a transport layer seems it would come closest.

On the other hand, If you simply want to obfuscate so sys admins can't readily read messages while doing maintenance and debugging -- rot47 is sufficient.

If your server is in the US and you are trying to have a solution that has some baseline protection for users when the US government and civil and criminal investigators make various administrative legal requests under the laws governing stored communications and message interception, E2EE fits the bill but is not 100% necessary (a more modest approach combining technical solutions including encryption and clear business practices along with a decent legal team is sufficient).

If you are (or feel you are) at war with your own government or some other national state actor or have some reason to worry that your messages will cause you embarrassment or other trouble if obtained illegally and somehow released -. first it'd be time to question whether or not you should be doing what your doing, and if so, you should consider that internet communications and message storage may not be the best mechanism to carry out business. But if you want to go there, a solidly designed and crafted E2EE solution is probably the only solution - but even with that it must be understood as an 'arms race' and if a 3rd party obtains your encrypted data and is that interested and has the resources, it is only a matter of time before the encryption will be broken (not to mention, trying to deal with all the other attack vectors including how to keep the unencrypted messages secure from "snooping" on your device - eg, making sure your smartphone keyboard isn't already sending your messages to 3rd parties already (btw, it probably is) or someone hasn't installed a keylogger on your desktop or tapped into your VGA/HDMI cable and installed a recorder or transmitter.

Just like physical security, data security has various degrees and each comes with a price tag. The proper solution mitigates the reasonable threats and ignores the unrealistic (though still potentially possible) threats realizing that if they actually happen, there will be far more urgent and important things to worry about than the porn your trying to hide from your partner.

El Dorado

Fri, 15 Mar 2019 22:44:18 +0000

Exactly. In Hubzilla we don't treat the site admin as an adversary. This is also primarily a multi-user web publishing system so we need to safely present multi-media HTML content, and we can't sanitise E2EE content. This throws you back into a text-only darkweb which isn't very social and most people reject it. They tend to prefer multi-media interactions and viewing their stream as a coherent stream on a single device. To protect against the site admin you'll need E2EE, but once you add E2EE, you really have to compose and view it offline. Files and media need to be encrypted offline before they are uploaded. Inline images won't work easily in this world unless they are embedded as binary blobs within the message.

Pseudo-E2EE (JS-based) *may* protect you from the site admin but you'll never really know for sure. On the bright side, with a bit of effort you can display multi-media in your normal stream (as our demo E2EE crypto app shows). So that's an important usability trade-off, and the main reason why the software provides this option.

When state actors started poking into my life (the Friendica IS episode a few years back) they didn't bother with any encryption. I found keyloggers everywhere. My work computer (on a private network behind a proprietary commercial firewall), my home computer. The wife's computer. The daughter's phone and tablet. We had every OS represented and they were all backdoored. The router was backdoored. I'm fairly certain my ISP was compelled to log my system activity.

As a project, we're interested in protecting you from commercial exploitation. We simply don't have the resources to protect you from the state. You're welcome to donate but currently the donations barely cover operational costs. They aren't providing a paycheck for anybody and we would need a pretty well financed team to start building a solution that is even safe from third-world state actors. I wish I could talk to you in detail about encryption because I have learned a lot over the years. I know where there are (serious) flaws in mainstream crypto technology. Unbreakable encryption isn't all that difficult, but the rest of the ecosystem is easier to hack and will nearly always divulge the secret at a much lower cost.

Ultimately it all boils down to the value of your information and the security discipline of everybody in the path of that information, and contrast that with the resources available to your adversaries. The one with the most resources usually wins, but you can get by for a while with scrupulous organisational discipline. If you want to protect your communications against state actors, I'd recommend hiring a consultant to your organisation that can find and secure all the weak links. Everybody involved will have to be trained in information security, really everybody. Once you do that, you'll probably find you don't even need crypto because you won't be using open source software on the public internet to transport your information.

So as Matt asked - who do you see as the adversary? And what is the value of your information? And how far are you willing to go and how much inconvenience are you willing to accept to protect it?

Sat, 16 Mar 2019 01:25:39 +0000

In Hubzilla we don't treat the site admin as an adversary.

At the same time, there some landscape changes - clearly not everyone is going to run their own hub. There is certainly going to be some level of centralization and people are going to entrust their data to "3rd parties" as hosting providers. So the hubmin/user relationship does need to be considered at least a "reduced trust" environment if not trustless.

Which leads to....

who do you see as the adversary? And what is the value of your information? And how far are you willing to go and how much inconvenience are you willing to accept to protect it?

So - as a hub admin considering offering service to 3rd parties who I may or may not know, I find myself thinking about it a little differently.

My comments below are most definitely U.S. centric with regard to the law, but I think a system built for this environment will fill several needs - even under the GDPR which requires certain assurances about security and privacy from 3rd parties.

Under the 3rd party doctrine regarding "privacy" in the U.S, data under my control as a 3rd party is subject to the rules of discovery in a court case or various types of law enforcement investigations. I'm not just talking about "state actors spying on people," but you also have the lawyers of a spouse in a fierce divorce case who can subpoena data from your (the hub admin's) server. Other civil and even criminal cases have the same issues - granted, serious crimes like terrorism or espionage are going to get the spooks involved -- but more mundane crimes - white-collar crime, as well as violent and non-violent crime, and even corporate espionage cases, etc. - are going to trigger investigative subpoenas from various law enforcement and other legal processes. As a hub admin, I'd very much like to be able to say, "well, I can give it to you, but you won't be able to read any of it because it's encrypted and the decryption keys are not 'discoverable' (i.e., they are not legally able to be requested)."

Basically, I want to "do right" by my users, treat what should be private as private and not make it easily discoverable/disclosable.

There is also a "forward looking" element. At the moment, CALEA (the US - Communications Assistance for Law Enforcement Act) does not apply to email and messaging providers. However, the FCC has already pushed to blur the line between INFORMATION SERVICE PROVIDERS and CARRIERS with recent rules (mostly targetting VOIP and other real-time interactive technologies). It will only be a matter of time before they push further.

Thus, MY current answer to the question, "Who do you see as the adversary?" is "The legal discovery process and LEO's who want to rope me into their investigations and use a platform that I 'run' to pry into areas which should generally be prohibited if not for the fact that this person is using software on my server."

A secondary answer is: In an environment where a commercial service may be running one or more hubs for customers - a mechanism to make sure that employees are unable to easily snoop and intercept messages not intended for them becomes important.

So for me (and I know this isn't all about me, but I think we need some "real world reasons" to start with this discussion) I see needs as follows:

Encryption for messages "at rest" (which I am defining as stored in the database and retrieveable through the "select" statement).

I believe that there is sufficient means already to deal with files "at rest" through the use of layered FUSE file systems and various administrative controls.

Encryption keys are sent TO the hub for use BY the hub - and so are not 3rd party information, but rather are 2nd party information and therefore (theoretically) not discoverable. Though, a mechanism that would not make the hub actually store the decryption key would be the best (something akin to the BitShares digital wallet was something I thought of a while ago in order to do some of this).

Anyway, there are various pieces here. And while I think I have a handle on the concepts, I'm pretty sure I'm at that stage where I know enough to delude myself into thinking me and my users are safe while leaving things WIDE open.

Knowing that the guy who knows this stuff best can't discuss it, I've reached out at one point to the EFF and back in January a brief note to D.J.Bernstein asking if maybe one of his grad students needed a project to work on (hey, it's worth a shot!) but so far, I have found nobody to talk about these issues with from both a technical and a legal frame of mind - and I don't have $$$ to be able to pay someone to talk with me.

In any event, that lays out a bit of the "real world" that I'm thinking about when it comes to these topics.

Notice that all of this falls short of E2EE. There is still the ability at certain points on the hub to see the clear text in order to apply filters and vet data - the idea here is to keep the number of those points to a bare minimum and, if possible, to not leave the "keys" on the server any longer than (and unless) absolutely necessary.

Sat, 16 Mar 2019 01:34:53 +0000

(BTW -- THANKS! Forcing myself to think through all of that helped clarify a number of things!)

El Dorado

Sat, 16 Mar 2019 04:52:50 +0000

Yes, but as has been pointed out time and time again, if the keys are on the server ever, the admin can see the content. Obfuscation in this manner actually does nothing to improve security and only kills performance (really kills it. Back when we did this with redmatrix it was the difference between a 2 second and 2 minute stream load for 40 conversation items, and the admin can still see everything if he/she wants to). If you have to do shenanigans to get keys from clients or through Diffie-Hellman zero knowledge algorithms from the client via JS you increase this time and also reduce the capabilities of the system and in most cases the admin can still follow the same steps and see the content. You also can't access anything via API unless the client was written with key discovery included - so for instance you'd have to throw out the Twitter API completely.

We've been down this road before. It turned out to be a dead end. You paid dearly in terms of performance to make it only marginally harder for the admin to snoop into your stuff. Ultimately this all needs to be P2P but the world isn't anywhere near ready to throw away server technology yet. This isn't stopping somebody from using E2EE when they need to, but retaining all the other advantages of a server-based platform for content which doesn't require a lot of protection. It's a balancing act.

Sat, 16 Mar 2019 12:01:13 +0000

Yes, but as has been pointed out time and time again, if the keys are on the server ever, the admin can see the content.

But this isn't the security consideration I'm primarily concerned with. If you store your data on someone else's server and expect their server to handle encryption/decryption - it's impossible that they can't access it somehow.

kills performance

Since I have no data, I have to take your word for it and I don't really doubt it, though there may be some mitigation of the performance hit that can be done now that wasn't available previously (eg., use local storage to keep copies of decrypted messages on the user's system and only retrieve/decrypt when needed - yes, this would take a huge rework of the user facing stuff, I know, but even some incremental work in this direction will improve performance in a non-encrypted environment - [not that I'm going to have time anytime soon to work on it... but it's a thought to throw out there in case someone else has the "chops" for it])

It's a balancing act.

Oh, it DEFINITELY is. And my "advocacy" here is not intended to be an argument in the least. Just an exploration and a continuation of the conversation in hopes that a new mechanism/idea may come out of it that hasn't been considered OR the possibility that the technical landscape has changed in some way (eg, local storage) that wasn't available when "we've been down this road before."

Back to something you said previously:

This is also primarily a multi-user web publishing system

I think this needs to be kept forefront as well. "Publishing" systems are by nature intended to make data visible. So not only is it a "balancing act," it is somewhat "against" the main premise of the software in some ways. Thus why, on the whole priority spectrum, this really probably shouldn't receive extensive effort unless there is a clear path forward and the benefits clearly outweigh the costs. But without these little forays and thought experiments, such paths will never be found or seen and opportunities may be lost.

hEARt PhoniX

Sat, 16 Mar 2019 14:19:17 +0000

I'd argue it's not against the main premise of the software. You could want to publish only and exclusively to a known audience. There are very good reasons for that all around. Among those: using Hubzilla as your business colaboration tool connecting either subsidiaries, coworkers or whatever.

Sat, 16 Mar 2019 15:26:38 +0000

You could want to publish only and exclusively to a known audience.

That's actually part of my interest in Hubzilla. But that doesn't require encrypted messaging.
OWA + "nomadic content" would be the preferred mechanism. Basically - don't send data to a server you don't control/trust. And if you're really picky - don't allow channels with clones on untrusted hubs either (but at that point, you may as well just "silo" your data and require separate authentication).

using Hubzilla as your business colaboration tool connecting either subsidiaries, coworkers or whatever.

You would presumably have "your own" hub (or sub-network of hubs) in that case - under your own administrative control a hub at each subsidiary, for instance).