Cryptography Stack Exchange
Cryptography Stack Exchange

Questions tagged [tokenization]

Tokenization is the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens.

Tokenization is the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens.

The list of tokens becomes input for further processing such as parsing or text mining. Tokenization is useful both in linguistics (where it is a form of text segmentation), and in computer science, where it forms part of lexical analysis.

11 questions
6
votes
2 answers

DRM simple clone

I have been trying to encrypt media(Video and audio content) from a nodejs server to a client like android or iOS. I heard of DRM but could not get a reliable implementation of DRM. I wanted to make my own implementation of protecting my content,…
Ceddy Muhoza
  • 163
  • 3
  • 7
5
votes
3 answers

How feasible is word-level frequency analysis over English (or any language)?

Say I have some black box which, given any English word, deterministically outputs a token for that word. Assume our black box is implemented using strong cryptography, i.e. the hardness of reversing a token to its word is reducible to some known…
pg1989
  • 4,736
  • 25
  • 43
2
votes
1 answer

JWT Common Practices

I had a few questions in regards to certain practices used by JWT developers. I'm relatively new to both encryption and JWTs and the context given is for developing a system on nodejs. How should verification work? Should I look at the JWT's…
Scetra
  • 23
  • 2
2
votes
2 answers

How do I verify a reset token if the token is hashed in DB?

I've been searching around the web for best practice how-to on secure password reset system (email with reset url), and the idea to hash the token in the database, which I did not implement at first, seems to be the the most secure way to store the…
Contra
  • 129
  • 1
  • 3
1
vote
1 answer

Question on Tokenization and the need to maintain a value to token lookup

I'm a newbie and an idiot to boot. I've a question in relation to tokenizing: is there a methodology out there that would allow you reidentify without using a token to value lookup? I'd have thought no but have a developer claiming something…
LarryTheLamb
  • 11
  • 2
1
vote
1 answer

Is it feasible to execute a range query over tokenized data?

Suppose we have a DB with n records on it, one of the fields is numeric and considered sensitive (e.g. ID number). If we use tokenization for said field, could we query all records between x and y? So far the only option I have seen depicted was…
Alvai
  • 63
  • 6
1
vote
2 answers

PAN authenticatable irreversible tokenization algorithm

I need to implement algorithm to create authenticatable irreversible tokens from PAN, without using any secrets. Is it possible? Is there any standards for that from associations? I was thinking about HMAC but it needs to distribute secret symmetric…
user1563721
  • 583
  • 4
  • 17
1
vote
3 answers

Base64 or hex to attach auth tokens to URL?

I just came across this question, but it does not really answer my main concern that I have regarding password tokens. To authenticate a client who is not logged in; I generate two random tokens and then use one as the DB selector, the other one as…
DevelJoe
  • 215
  • 2
  • 8
0
votes
2 answers

Understanding token security

I am having some problems, cryptographically speaking, digesting this information. There is a table that seems to say that tokens provide "end to end security" (I suppose we understand different things by that). But my main issue is with…
user1156544
  • 129
  • 6
0
votes
1 answer

Token generation using Mac?

I'm writing a web service that gives a unique token per client connection. And the following are the desired properties. I'm assuming server's issued token db is lost and untrustable client presents the token to the server. Server should be able…
Kibeom Kim
  • 3
  • 1
0
votes
0 answers

Secure algorithm for calculating token and comparison

I am trying to define algorithm that would create a irreversible token from IDs, which are 10 digit information, and then would be compared on two different end points. Between end points, only sending tokens is allowed, no IDs can be send from one…
user1563721
  • 583
  • 4
  • 17