NCPDP is a quirky legacy format that would feel alien to any modern-day developer or an IT person.

For example, it uses non-printable separators and non-mnemonic two-character field identifiers; it relies on an obscure way to encode numbers with decimal points; it uses a mixture of variable and fixed length fields.

Nevertheless, the format is still widely used in the pharmacy space.

We published several posts to help anyone dealing with this pesky format quickly get up to speed:

• NCPDP introduction in plain English. If you know nothing about the NCPDP format, start here.
• NCPDP separators. This post explains the “non-printable” separator characters used to denote fields and records.
• NCPDP headers. NCPDP uses fixed-length transaction headers. The problem is there are two different versions of the same header. This post explains it all.
• Interactive example of a pharmacy claim in NCPDP format. This is an example of a typical pharmacy claim with a field-by-field explanation.

Finally, we provide a completely free NCPDP viewer and file reader to help you make sense of your NCPDP files.

There are several flavors of EDI; X12 EDI is used in the US. This post provides a good introduction to X12 EDI.

DataPower gateways have a B2B feature that enables the processing of X12 EDI.

We’ve developed several free tools to help deal with X12 EDI, especially when it comes to healthcare-specific EDI transactions. The tools are available from our sister website, Healthcare Data Insight, along with many examples and tutorials.

Our tools also provide support for the NCPDP telecommunication format. NCPDP is the format (and the standard) used for pharmacy claims and related transactions in the pharmacy space. The NCPDP format conceptually resembles X12 EDI but uses completely different separators and data elements.

Here is our list of free EDI and NCPDP tools:

• X12 EDI viewer and file reader. The viewer converts any EDI input into a user-friendly format.
  It provides a special business view (without EDI artifacts) for 837 (healthcare claim) and 835 (payment) transactions.
  The viewer automatically decodes CPT, ICD-10, NDC, and other healthcare codes so that you can immediately see a code’s description. You can find multiple examples of the viewer in action here.
• NCPDP viewer and file reader. The NCPDP viewer shares the user-friendly UI with our X12 EDI viewer and provides similar functionality adapted for the NCPDP format.
• API to convert 837 and 835 X12 EDI transactions. This free API consumes X12 EDI text and returns a claim or a payment business object. The schema of business objects is intuitive; it requires no knowledge of X12 EDI.
• API to convert text in NCPDP telecom format into JSON.

We also provide a free healthcare code lookup tool to make it easy to decipher healthcare codes, such as CPT, ICD-10, NDC, UB-04, and others.

All DPBuddy’s dependencies (third-party libraries used by DPBuddy) have been refreshed.
This includes removing and updating libraries with security vulnerabilities.

DPBuddy has been tested with all the recent versions of Java, up to Java 19. Java 8 is the lowest-supported Java version.

We have thoroughly tested DPBuddy 3.5.2 against the most recent DataPower firmware releases, including 10.5.

• Inventory of all your certificates, keys and password aliases across all of your appliances and domains.
  In addition to DataPower, the inventory includes certificates from TLS endpoints.
• Reports in Excel and text format
• Certificate deduplication — find all places where a given certificate is used
• See audit records of all changes — who changed what when
• See how each of your crypto objects is used by other DataPower objects
• See expiration for passwords/password aliases (in addition to certificates’ expiration)
• Ensure compliance and best practices: identify self-signed certificates, weak keys/algorithms, unapproved signers
• Get alerted on certificate expiration, invalid signatures, revocation, policy violations

Here are examples of various reports generated by the “listCrypto” command.

Certificates with their audit records. You can see that the same cert (lines 2-5) is deployed multiple times under different file names.
The cert at line 5 was downloaded directly from the endpoint.

Domain          Object Name    File                     Subject CN           Alt Names                           Exp in Days Key Alg  Issuer                              Serial                                Changed On Changed By
dpbuddy-samples revoked_chain  cert:/revoked_chain.pem  revoked.badssl.com   revoked.badssl.com,www.revoked.bads         473 RSA-2048 DigiCert SHA2 Secure Server CA      4578095623763233818958520798617405692 6/20/20    aananiev
dpbuddy-samples myarch_chain   cert:/myarch_chain.pem   myarch.com           myarch.com,www.myarch.com                   493 RSA-2048 Go Daddy Secure Certificate Authori 11510493113533735686                  6/20/20    aananiev
dpbuddy-samples myarch         cert:/myarch.pem         myarch.com           myarch.com,www.myarch.com                   493 RSA-2048 Go Daddy Secure Certificate Authori 11510493113533735686                  6/20/20    aananiev
dpbuddy-samples myarch.com_443 cert:/myarch.com_443.pem myarch.com           myarch.com,www.myarch.com                   493 RSA-2048 Go Daddy Secure Certificate Authori 11510493113533735686                  6/20/20    aananiev
https://myarch.com:443                                  myarch.com           myarch.com,www.myarch.com                   493 RSA-2048 Go Daddy Secure Certificate Authori 11510493113533735686
dpbuddy-samples local-app      cert:/local-app.pem      local-app            local-app                                   326 RSA-2048 Local CA                            1589417790                            6/20/20    aananiev
dpbuddy-samples local_app      cert:/local_app.pem      local-app.com                                                    266 RSA-2048 local CA                            1584288750                            6/20/20    aananiev
dpbuddy-samples self_signed    cert:/self_signed.pem    localhost                                                       -523 RSA-2048 localhost                           9699314724490867386                   6/20/20    aananiev
dpbuddy-samples keypair-1      cert:/keypair-1.pem      test-key-1                                                      -278 RSA-2048 test-key-1                          1537189500                            6/20/20    aananiev

Results of the certificate compliance verification; reported in the “Problems” column

Object Name    File                     Subject CN         Exp in Days Key Alg  Issuer                              Serial                                Changed On Changed By Problems
revoked_chain  cert:/revoked_chain.pem  revoked.badssl.com         473 RSA-2048 DigiCert SHA2 Secure Server CA      4578095623763233818958520798617405692 6/20/20    aananiev   Duration exceeds,Ocsp revoked
myarch_chain   cert:/myarch_chain.pem   myarch.com                 493 RSA-2048 Go Daddy Secure Certificate Authori 11510493113533735686                  6/20/20    aananiev   Duration exceeds,Blacklisted issuer
myarch         cert:/myarch.pem         myarch.com                 493 RSA-2048 Go Daddy Secure Certificate Authori 11510493113533735686                  6/20/20    aananiev   Duration exceeds,Blacklisted issuer
myarch.com_443 cert:/myarch.com_443.pem myarch.com                 493 RSA-2048 Go Daddy Secure Certificate Authori 11510493113533735686                  6/20/20    aananiev   Duration exceeds,Blacklisted issuer
local-app      cert:/local-app.pem      local-app                  326 RSA-2048 Local CA                            1589417790                            6/20/20    aananiev
local_app      cert:/local_app.pem      local-app.com              266 RSA-2048 local CA                            1584288750                            6/20/20    aananiev
self_signed    cert:/self_signed.pem    localhost                 -523 RSA-2048 localhost                           9699314724490867386                   6/20/20    aananiev   Expired,Self signed
keypair-1      cert:/keypair-1.pem      test-key-1                -278 RSA-2048 test-key-1                          1537189500                            6/20/20    aananiev   Expired,Self signed

Certificate checks are configured in “crypto.conf” as following:

expiration: {
    days: 30
}
duration: {
    days: 365
}
selfSigned: {
    allow: false
}
sigVerification: {
    failIfNoIssuer: false
}
ocsp: {
    enabled: true
}
pubKey: {
    minSize: {RSA: 2048, EC: 256}
    allowedAlgs: [RSA, EC]
}

Private keys with usages:

Object Name                  File                                   Usage
dp-myarch-key                cert:/dp_myarch.key                    CryptoIdentCred:dp-myarch-cred
datapower.myarch.com_privkey cert:/datapower.myarch.com_privkey.pem
local-app_privkey            cert:/local-app_privkey.pem
local-app-ss_privkey         cert:/local-app-ss_privkey.pem         CryptoIdentCred:datapower-local-ca, SSLServerProfile:dp-myarch-server, XMLFirewallService:OktaResourceOwnerFlow
sslserver                    cert:/sslserver-privkey.pem            CryptoIdentCred:sslserver, CryptoProfile:sslserver, SSLProxyProfile:sslserver, HTTPSSourceProtocolHandler:https_5041, MultiProtocolGateway:oauth-mpgw-rs

Password aliases with expiration and usages:

Location Domain          Object Name         Exp in Days Changed On Changed By Usage
dev      dpbuddy-samples self_signed_key-pwd             6/20/20    aananiev   CryptoKey:self_signed_key
dev      dpbuddy-samples oracle-pwd-alias             83 3/16/20    aananiev
dev      dpbuddy-samples keypair-1_key-pwd               6/20/20    aananiev   CryptoKey:keypair-1_key
dev      dpbuddy-samples service1-pwd-alias          -66 3/16/20    aananiev
dev      dpbuddy-samples service2-pwd-alias          -97 3/16/20    aananiev

The report in Excel format provides additional information, here is the snippet of the report:

More details are available in the documentation.
If you’re interested in fully automating your certificate and key management, please let us know.

Can we evaluate security of a website or a digital signature based on its public key without any access to the private key?

Turns out that we can, at least, to a degree.

First, we need to understand that it’s the combination of the algorithm and the key size that defines the “strength” of the key. The most widely used public-key algorithm today is RSA. Elliptic Curve (EC) algorithms (that are truly superior to RSA) are quickly gaining ground and taking the second place.

As per NIST definition:

Cryptographic algorithms can provide different “strengths” of security, depending on the algorithm and the key size used (when keys are required by the algorithm). Security strength is a number associated with the amount of work (i.e., the number of operations) that is required to break a cryptographic algorithm or system.

The same document provides a handy summary of the strength of various popular algorithms/key sizes.

Roughly speaking, RSA-2048 (the de-facto today’s standard) is equal in strength to EC-255.

What about the relationship between the public key and its private key?

For both RSA and EC the size of the public key depends on the size of the private key. This dependency is less direct for EC and pretty much one-to-one for RSA, at least for TLS certificates.

This, of course, is only part of the story. For TLS, asymmetric cryptography is used only during the handshake phase of the TLS negotiation. As part of the handshake, the two parties agree on the symmetric key, which is actually much smaller than the asymmetric key. The algorithm and the size on the symmetric key depend on the supported cipher suites.

For example, a TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 suite shows that the symmetric AES-256 key will be used for “bulk encryption”, that is, as the session key for encrypting the actual data.

More detailed explanation can be found here.

The public/private key strength is even more important for application-level protocols, like OAuth2/OpenID Connect. In this case, public-key cryptography is used for signing/verifying the signature of a JWT token (and, optionally, for its encryption).

JWT/OAuth2 uses a slightly different nomenclature for key sizes, specifically, the size is in bytes versus bits, e.g., RS256, which is the same with RSA-2048.

The key’s size/strength is also critical for any kind of data at rest encryption or in a situation when a digitally signed document can be stored for a prolonged period of time.

The size of a key is deemed sufficient only for some period of time until the hardware becomes more powerful to allow for cracking the crypto algorithm with brute force or other approaches. This is the reason why the key strength has been marching upwards for many years.

RSA-2048 is considered secure until 2030, but it could be just a guess, given advances in quantum computing.

Imagine a digitally signed PDF contract (e.g., a real estate deed) that has to be stored for potentially tens of years. Currently, Adobe digital ID utilizes RSA-2048, so what if this algorithm becomes vulnerable five years from now?

The same problem applies to encrypted backups, disk encryption, and database encryption. In general, these use cases require stronger keys than the ones used for inherently transient TLS sessions and OAuth2 tokens.

• Deployment from standalone files in various formats (PEM, DER, PKCS8, etc.), encrypted and unencrypted.
• Deployment from Java keystores/truststores in various formats (JKS, PKSC12, etc.). You can specify a list of aliases to deploy a subset of certs/keys from a keystore.
• Deployment directly from TLS endpoints to DataPower.
• Automatic deployment of issuers/CA certs. DPBuddy can also download the issuer from the certificate’s AIA extension if exists (all certs issued by known CAs will have that extension).
• Auditing of all changes to crypto objects directly on DataPower. You can see who changed what when using DPBuddy’s crypto reporting task.
• Keystores and key passwords can be stored encrypted in DPBuddy’s conf file or provided directly on the command line.
• Deployment is automatically validated to make sure all crypto objects and password aliases are up.
• Crypto Identity Credential objects are created automatically for cert-key keypairs from a keystore.

We’ve also developed a framework for integrating with your Key Management System of choice, such as Hashicorp Vault or AWS Key Management Service.

DPBuddy copies keys/cert files to DataPower (as PEM files) and creates DataPower crypto objects. The names derived either from filenames or from the names (aliases) in the keystore.

DPBuddy automatically determines if the source is a key or a cert and creates the crypto objects of the appropriate type.

All certs are de-duplicated to make sure that each unique cert is only deployed once. This comes handy when, for example, the same CA’s cert can be found in multiple keystores or files.

If a cert/key is password-protected, the password alias object is automatically created on DataPower with the same password.

Here are some examples.

Deploy certs and keys from various stand-alone files:

dpbuddy deployCrypto -dir crypto_files -includes "*.pem *.cer *.key *.pkcs8" -excludes "myarch.pem" -passwords "self_signed:changeit"

Deploy certs and keys from the keystore:

dpbuddy deployCrypto -file crypto_files/test_keystore.jks -ksNames "local-app, keypair-1" -passwords "test_keystore:changeit,keypair-1:changeit"

Deploy directly from an endpoint (including the issuer/CA certs)

dpbuddy deployCrypto -endpoint myarch.com

Deploy only the certs (and their issuers) that match certain domain name (e.g., “myarch.com”) in CN or in the alternative names extension.

dpbuddy deployCrypto -subjects myarch.com -dir crypto_files -includes "*.pem test_keystore.*" -passwords "-passwords "\${crypto.passwd}" -keys false

“${crypto.passwd}” variable points to the following line in dpbuddy.conf:
crypto.passwd: “test_keystore:ENC{+MmRqtwXqYFpbpk9r3NTfrxMXR9m21xT}, keypair-1:ENC{+MmRqtwXqYFpbpk9r3NTfrxMXR9m21xT}”

More details are available in the documentation.

If you’re interested in fully automating your certificate and key management, please let us know.

On mac os you even get a warning that a website presents a non-standard certificate when you try to open the cert in Safari or Chrome (but not in Firefox).

First of all, you must have the Subject Alternative Name (SAN) extension, this extension must contain DNS names of all the domain names the certificate was issued for. Browsers no longer trust the “CN” of the subject field.

Your certificate must have a “Basic Constraints” extension marked as critical and specifying that the subject is not a CA.

Your certificate must also have a non-critical “Extended key usage” extension specifying “TLS Web Server Authentication” (if your certificate is used by a server).

Another requirement is to provide a “Key Usage” extension (marked as critical) allowing for “Digital Signature” and “Key Encipherment”.

Finally, and this is just common sense, your certificate’s validity must be limited to 825 days (the shorter the better) and the RSA’s minimal acceptable key length is 2048.

Internal CA is really just a self-signed cert (keypair). Make sure to use proper extensions when creating it. At the very least, specify “Subject is a CA” in “Basic Constraints”.

Do not set CN in the subject so this keypair can never be used as a cert for actual domains.

CAs are allowed to have a long validity period, remember that you will need to reissue all the certs once the CA’s cert expires. However, we recommend limiting it to 5 years since you’d want to automate your cert provisioning anyway.

Keep your internal CA keypair in a separate keystore (Java) or in a completely separate place (ideally, in your centralized secrets/key manager). Never place the CA’s key in the same keystore with the certificate issued by this CA. Of course, the X509 of the CA could be widely distributed.

You can now start generating end-entity certs.

Create a new keypair (a self-signed cert) in the Keystore Explorer.

Your cert must have certain extensions, otherwise, it will be considered invalid, especially by mac OS browsers.

At the very minimum, it must include “Basic Constraints”, “Extended Key Usage”, “Key Usage”, “Subject Alternative Name” with the DNS names of your (internal) domains.

Note that browsers no longer trust the “CN” field and the “Subject Alternative Name” extension is a must.

The validity period cannot exceed 825 days, but the shorter the better. For internal certs, we recommend 90 days.

Now right-click on the new keypair and select “Generate new CSR”, save it to a file.

Now open the keystore with your local CA, right-click on the CA keypair and select “Sign CSR”. Remember that the private key is used for signing, whereas the signature can be validated with the CA’s public key (this how the whole certificate chain validation works)

Save the result in a p7r file which is a special format for CA’s responses. Do not forget to click on “Transfer extensions”, otherwise all the extensions in the original cert and in the CSR will be lost.

Now go back to the keystore with your self-signed cert keypair and select “Import response from CA”. Select the p7r file. You should now see your internal CA in the chain. Make sure that all the extensions are in place.

Save the keystore.

You can now import your internal CA into various browsers and when you navigate to your internal site, you should not see any security warnings.

To minimize the effort, self-signed certificates are often issued for several years, which, of course, compromises security.

An internal Certificate Authority can be used instead to avoid the issue of updating expired self-signed end certificates.

An internal CA can be issued for multiple years, so there is no need for frequent updates across all clients.

An internal CA is really not much different from an “official” CAs like DigiCert. DigiCert would simply verify domain ownership before signing the certificate request. Once this is done, it would signs our certificate request thus creating a valid X.509 cert. The signature (usually using SHA256WITHRSA) is stored in a separate field in the X.509 cert as defined in rfc5280.

If needed, an internal CA could conduct a more in-depth verification of the request, including obtaining information on what kind of service the new certificate will be used for.

It can also mandate various extensions on key usage thus making the cert more secure. Of course, certificate duration can be reduced to six months or less; there is really no reason why a certificate should be valid for 12 months.

Establishing a CA-level trust is convenient, however, it also requires to have a revocation process in place. This process could be based on good old CRL or OCSP.

You must have to have an ability to revoke a compromised cert, otherwise, the certificate and its private key can be used for nefarious purposes if the server is compromised.

There are many ways to implement OCSP/CRL, we offer a mechanism built into our DPBuddy/SecScan product.

Please make sure to properly guard the private key of the internal CA. It has to be stored separately from any certificates that it issues since it allows us to issue a cert for any internal domain.

The X.509 counterpart of the internal CA should be freely distributed to all trust stores of all the components of the system as well as to the browsers of the end-users of the application.

You you can use openssl CA command to sign internal CSRs thus becoming your own certificate authority.

Public and private keys can be stored in several different types of files. Each of these types can have its own encoding. The overall format of a file can be quite complex. It is important, however, to understand the purpose of these formats, and how they’re used.

This document can be used as a primer for understanding these file/encoding formats.

The actual structure (objects and fields) of public/private keys, including X.509 certificates, is specified in various RFCs using the ASN.1 notation.

For example, an RSA private key contains the following fields:

RSAPrivateKey ::= SEQUENCE {
  version           Version,
  modulus           INTEGER,  -- n
  publicExponent    INTEGER,  -- e
  privateExponent   INTEGER,  -- d
  prime1            INTEGER,  -- p
  prime2            INTEGER,  -- q
  exponent1         INTEGER,  -- d mod (p-1)
  exponent2         INTEGER,  -- d mod (q-1)
  coefficient       INTEGER,  -- (inverse of q) mod p
  otherPrimeInfos   OtherPrimeInfos OPTIONAL
}

The most widely used format is X.509 and it’s full syntax is defined by the RFC5280. X.509 provides the support for the “chain of trust” to verify a public key, as well as various extensions, primarily concerning the key’s usage. RFC5280 also documents formats for CSR, CLR, etc.

PKCS #8 specification defines the structure of private keys (PKCS stands for Public-Key Cryptography Standards).

These specifications only stipulate the structure (fields and objects), we still need to decide how to “encode” these fields when we want to save them on disk.

Security considerations aside, it would be nice if everything was stored in some sort of a structured format that we’re all used to, such as JSON or YAML, but this not the case for the majority of crypto formats, with the exception of JWK as explained later.

“DER” encoding, which is a variation of a more general X.690 encoding, is the de-facto standard for encoding ASN.1 structures. DER is a purely binary format, DER-encoded content can’t be inspected with a text editor. The basic idea of DER is to represent each field as a type+length+value triplet.

DER format is the one used for sending certificates over the wire — e.g., when a server presents its certificate to the client as part of the TLS handshake, the certificate bits are serialized using DER (theoretically, a certificate can be stored in a completely different format on disk).

DER can be further encoded using Base64, which creates an ASCII text that could be copied-pasted, sent over email and so forth. Base64-encoded DER files are usually saved with the “pem” or “crt” extension. PEM files also contain a header and footer to give an idea of the content of the file (sort of poor man metadata). X.509 files use “—–BEGIN CERTIFICATE—–“/”—–END CERTIFICATE—–“, but various other headers/footers can be used as well, such as “—–BEGIN PUBLIC KEY—–“/”—–END PUBLIC KEY—–“. In this last case, the file contains only the public key information and none of the additional X.509 fields.

Private keys can be saved in the PEM format as well, “—–BEGIN PRIVATE KEY—–“/”—–END PRIVATE KEY—–” is used to denote such files. All these headers as well the detailed PEM-encoding rules are documented in this specification.

ASN.1/DER/PEM is mostly used for TLS implementation and whenever X.509 certificates are involved.

Other public-key cryptography implementations can use different formats. Irrespective of the format, the underlying implementation and algorithms remain the same. E.g., an RSA public key must contain modulus and exponent fields, the only question is how to pass these fields around or how to store them in a file.

OAuth2 and JWT use JSON to exchange public/private keys.
Here is an example of a public key in the JWK format used by OAuth2 authorization servers:

{
    "kty": "RSA",
    "alg": "RS256",
    "kid": "oViynWdKmd9m43BihjrQH9bHlp22fto0Nu-zwaBzUAs",
    "use": "sig",
    "e": "AQAB",
    "n": "q8BD_0q9JQRnpZ5vLnBMEA03nUWmxE56nGvKFY8K0fOAHojFPExI0Il67NEv6TCPZaXiifT5p9N9DIQl-JaWNaQmDCvd5Hbeugqn05QGJ14E_ghTXA6iXsONnavri5qlgc5rPmAS9zkm755ID7mHnuskEMXJy929LlxFKHzDRTkN8Lf1hSVXG8Mdy0f1QW-01VNRE8ZW0Ar5vLLuGrDb8bg9fCZXA6CK7oVJHXzo6ajIgzpa86kpdvWOhhtYPCL9P9wNjt4kfX3LBb6_sl9s8lI0C0OWtoMyNtAbE4wFc08o0ZsW1UGQin5eFFBuH_zbaPwc7wvYw40bBw35U_V9Sw"
}

SSH keys (the ones that usually start with “ssh-rsa”) use their own form of encoding documented in the RFC4253 — each component of a key is stored as length+data and the entire key is Base64 encoded. Note that this is different from DER encoding, which is type+length+value. This post provides all the details.

It would obviously be a good idea to somehow encrypt and password-protect private keys. There are several formats that can be used for this purpose.

PKCS #8 files (usually encoded as PEM) files can be encrypted with a passphrase and various cyphers, in which case these file start with “—–BEGIN ENCRYPTED PRIVATE KEY—–” header.

The most widely used format for storing keys and certificates in an encrypted format is PKCS #12, defined by RFC7292. It can be used for storing certificates, public/private keys, and even arbitrary passwords. These files have “p12” or “pfx” extension (“pfx” is a PKCS #12 predecessor). Modern versions of Java use PKCS #12 as the default keystore format, although these files can still have the legacy “jks” extension.

Internally, PKCS #12 uses DER/ASN.1 structures for storing certificates and private keys.

For more recommendations on how to properly secure PKCS #12/JKS files, please refer to our keystore best practices post and the e-book.