It’s been a long time coming. We know we’ve been somewhat silent about all the work we’ve been doing for the past several months… but that’s because we’ve been hard at work for almost 18 months on Kazoo 4.0. Now we’re ready to share this great milestone with you!As part of this release, we’re trying to do a better job communicating changes and release dates. This email represents a “heads up” on all the new major features, bug fixes and backend improvements coming out in Kazoo 4.0. Open-source users should feel free to begin testing the new changes in 4.0 now (some of you already have). We are eager to hear your feedback and squash any bugs you may find! Make sure to include the specific build/package number you are on. Also, for those waiting for installation instructions that are step-by-step, we’re still polishing those but will release those as soon as they are ready, hopefully before KazooCon.
We will go into full detail about the release on stage at KazooCon, so if you haven’t already purchased your ticket, we encourage you to check it out – www.kazoocon.com!. In the meantime, here’s what to expect with the 4.0 rollout. If you have questions or want to discuss these items more click here, and check out our 4.0 announcements!
BACKEND
Functional and Performance
Couch Connector – Enhancement
What’s New:
We have made a major enhancement which lays the framework to allow Kazoo to connect to more than one database at a time, and to connect to different types of databases. This framework is utilized by other new features coming out, enabling new functionality and control over your database architecture. This also prepares Kazoo to officially use the upcoming CouchDB 2.0 software, which has a number of bug fixes and improvements.
How does it impact you:
This change should be transparent unless you are a developer, in which case you’ll need to review the new modules for Couch Connectivity (previously called couch_mgr, now known as kazoo_data library).
Couch Connector – Data Plans
What’s New:
Data plans allow you to configure where your system’s data is stored. It utilizes the new Couch Connector framework for connecting into multiple databases. It then allows you to setup “plans” where you can split data across different databases by data type (i.e. voicemail, call recording, etc.), by user (i.e. user A is on database A, user B is on database B), by zone (i.e. user C is in the New York cluster and user D is in the San Francisco cluster) and so on.
How does it impact you:
If you wish to take advantage of data plans to use alternate databases, you will need your Amazon, Google Drive or BigCouch cluster credentials/authentication keys. This is often in the form of OAuth authentication/permission. A UI component and APIs are not yet available for configuration, but you can follow the instructions on our documentation page to configure this manually and try it out.
External Storage Engine – Amazon S3
What’s New:
We have built support to utilize Amazon S3 as a storage provider (in conjunction with the new Couch Connector features). The engine allows you to configure a different Amazon S3 account and folder for each type of item stored, on a per-user, per-account, per-reseller or system-wide basis.
How does it impact you:
Kazoo Admins and Resellers will need to decide on a strategy for signing up and maintaining Amazon S3 accounts for either all customers, or a single sign-up that has permissions across their customers, or similar variations. In addition, customers are solely responsible for managing the keys and credentials for the Amazon S3 service and inputting them into Kazoo.
External Storage Engine – Google Drive
What’s New:
We have built support to utilize Google Drive as a storage provider (in conjunction with the new Couch Connector features). The engine allows you to configure a different Google Drive account and folder for each type of item stored, on a per-user, per-account, per-reseller or system-wide basis.
How does it impact you:
Kazoo Admins and Resellers will need to decide on a strategy for signing up and maintaining Google Drive accounts for either all customers, or a single sign-up that has permissions across their customers, or similar variations. In addition, customers are solely responsible for managing the keys and credentials for the Google Drive service and inputting them into Kazoo.
Number Manager Rewrite
What’s New:
The number management sub-system within Kazoo has been re-written to massively improve performance and stability for accounts which have 10,000 phone numbers in a single account, and for systems that have a collective total of over 1,000,000 numbers. The performance improvement will be perceived when doing number operations, especially bulk operations such as importing batches of numbers or deleting numbers.
How does it impact you:
Customer accounts will be migrated automatically the first time a phone number is modified (added/deleted/updated) within a single account. The automation will change the structure of the phone number documents within an account and on the global number database. This strategy ensures backward compatibility with the old structure, avoids risk at doing a mass-migration and avoids downtime. Please note that once an account has been upgraded to the new format, you can not downgrade to v3.xx of Kazoo.
Performance – Conditional Document Change Suppressions
What’s New:
Every time a document is updated in Kazoo, we publish a message that a document has been created/deleted/updated. We use these for self-flushing caches and Webhooks, etc. However, for certain notifications, they’re not useful and yet we’re still publishing them. Now, in the Erlang code and in system_config, you can select which Doc Update/Change notices go out to AMQP and which are suppressed (to avoid chatter on the message bus).
How does it impact you:
If someone has created a custom app that uses document change notices and is reliant on particular messages Kazoo produces, the customer must validate that those messages are still published and their app behaves the same way after upgrading to 4.0.
Performance: Cache Optimization Work
What’s New:
Every cache used to have an ETS table. We cache pointers, watchers and the object itself in the same cache. Because all three items were in the same cache, sometimes the cache itself would back up. We’ve changed Kazoo cache to comprise of three ETS tables – one for pointers, one for watchers and one for objects themselves. This makes it so we can reference everything by key and not search the caches anymore. This makes EVERYTHING much faster on large systems.
How does it impact you:
The cache changes have a dramatic performance impact on clusters with over 10k phones.
Voicemails Moved to Modb
What’s New:
Voicemails used to be stored in the same location as configuration settings. Accounts older than 12 months would often get too big, thus this became a design issue. Voicemails have now been redesigned so that the messages are stored in a monthly database that can be purged later.
How does it impact you:
All new voicemail messages goes into modb. All Kazoo Administrators need to migrate their voicemail messages from 3.22 to modb.
Account-Level Debug Mode
What’s New:
On highly active systems the default logging strategy is too busy. With this change the default log level can be reduced. Accounts experiencing trouble can have their individual levels increased for debugging.
How does it impact you:
There is no preparation required to take advantage of this improvement.
iBrowse Removed
What’s New:
In order to issue HTTP requests Kazoo and some of the Kazoo dependencies used a third party library called iBrowse. This library did not maintain backward compatibility so there wasn’t an upgrade path readily available. This resulted in Kazoo being stuck with a number of significant bugs in iBrowse which occasionally impacted call processing. In 4.0 iBrowse has been completely removed and a more stable HTTP client sanctioned by the core Erlang team is now utilized.
How does it impact you:
This upgrade prevents customers with CNAM lookups or heavy database usage from experiencing a partial outage. In addition, developers who previously coded using iBrowse should now switch to using kz_web module.
Data Structure
Support for over 1,000 numbers on a single account
What’s New:
Phone number storage has been enhanced to support a large quantity of numbers on a single account.
How does it impact you:
There is no preparation required to take advantage of this improvement.
Support for resellers to have their own phone number inventory
What’s New:
Resellers can now manage their own inventory of numbers.
How does it impact you:
There is no preparation required to take advantage of this improvement.
Ability to search the number inventory based on carrier as well as search for additional inventory
What’s New:
Massively enhanced the number search reservation and provisioning capabilities to support private number inventories and an infinite amount of secondary providers.
How does it impact you:
There is no preparation required to take advantage of this improvement.
Configuration
Kazoo core config file cleanup
What’s New:
The three global Kazoo configuration files in /etc/kazoo have been moved to /etc/kazoo/core.
How does it impact you:
After installing kazoo-config-core you will need to move the three files (which will be marked as .rpmsave) into /etc/kazoo/core.
Dispatchers list
What’s New:
Kamailio is responsible for distributing and load-balancing SIP packets to all nodes in a cluster. This distribution is configured with a database engine that tracks which nodes are in the cluster. In 4.0, we have upgraded to a more performant configuration engine. This change also corrects memory consumption issues and allows Kazoo to query the configured dispatchers list easily. Because of this change, the Kamailio dispatcher file format has also changed. It is now consistent with the formatting used by other files in the dbtext folder.
How does it impact you:
After installing kazoo-config-kamailio you will need to covert /etc/kazoo/kamailio/dbtext/
Management
Sup Commands
What’s New:
Our System Utility Program (SUP) utilized commands which used our old product name, Whistle, or wh in some cases for short. Beginning with 4.0 we have changed these commands to be consistent with the name ‘Kazoo’. Commands that contained wh have been shortened to k (such as kapps_controller). Additionally, the ability to use the -h argument and connect to machines remotely via SUP has been removed.
How does it impact you:
If you were using sup remotely you will no longer be able to and you will have to adapt any scripts that utilize sup to match the new command names. System admins will need to learn the new sup commands.
Requiring upgrade to CentOS7
What’s New:
CentOS6 is scheduled to stop receiving full updates Q2 of 2017. to ensure the highest level of reliability of our platform, 2600Hz will require CentOS7 so that Kazoo Administrators can continue receiving updates and security fixes in a timely manner.
How does it impact you:
The transition to CentOS7 also introduces the use of systemD which admins should prepare for.
Packaging
New RPM Repo
What’s New:
In order to support both Debian and CentOS, we needed to rebuild our packaging and automated release systems. However, we realize some users will still need access to the older packages for a while. To achieve both goals, we’ve created a new package repository for 4.0 packages (for both Debian and CentOS) but will also retain the old package repository for some time to come. Anyone moving to 4.0 should be aware of the new repository.
How does it impact you:
Kazoo Administrators will want to make sure they remove the old repo file and install the new one so the right packages are used.
Shipping both Debian & CentOS packages.
What’s New:
2600Hz is preparing to support both Debian and CentOS packages starting with Kazoo 4.0. We will continue supporting ONLY CentOS for paid clients as we start testing the feasibility of moving to Debian reliably.
How does it impact you:
Feel free to use either Debian or CentOS.
Freeswitch 1.6
What’s New:
2600Hz will stop maintaining custom FreeSWITCH packages. All patches we have done have been back-ported into FreeSWITCH 1.6. We will now utilize FreeSWITCH’s repository for all packages. This allows us to stay up to date with the developments in the FreeSWITCH project, including new features and bugs.
How does it impact you:
Make sure you use FreeSWITCH’s packages going forward and have their repo on your system.
Kamailio 4.4
What’s New:
2600Hz will stop maintaining custom Kamailio packages. All patches we have done have been backported into Kamailio 4.3. We will now utilize Kamailio’s repository for all packages. This allows us to stay up to date with the developments in the Kamailio project, including new features and bugs.
How does it impact you:
Make sure you use Kamailio’s packages going forward and have their repo on your system.
Individual Kazoo Apps
What’s New:
We will now package and ship all Kazoo applications as individual packages. This allows Kazoo Administrators to install, maintain and upgrade packages individually rather then ship all items together, and better exposes the modular capabilities of our system.
How does it impact you:
These benefits will be automatically realized when you upgrade to 4.0.
Requirement for Erlang 18 (not 19!)
What’s New:
We now require Erlang 18 (not 19!) for running Kazoo.
How does it impact you:
These benefits will be automatically realized when you upgrade to 4.0.
Erlang Releases-new packaging system
What’s New:
Kazoo is now an executable release, utilizing the Erlang release/package system. This means that the correct version of Erlang’s binaries comes with our software.
How does it impact you:
You no longer need to worry about what version of Erlang is installed on the machine. Kazoo will start with the correct version that is shipped with the Kazoo core. This behavior is similar to an executable binary.
International Language Packs
What’s New:
Kazoo now ships language prompts (sound files) as separate packages, per language. This allows you to install one or more languages of your choosing on your system and avoid installing extra languages you don’t need.
How does it impact you:
These benefits will be automatically realized when you upgrade to 4.0.
First compilation now requires internet connectivity to download dependencies
What’s New:
The first time you compile Kazoo applications, Kazoo will automatically attempt to download dependencies via the internet.
How does it impact you:
This means users who are compiling modules in Kazoo need to make sure they have internet access the first time they do this.
Call Recordings
Call Recordings
What’s New:
Call recordings existed in v3 but required additional effort from resellers to handle storage of the recordings. Most users found this cumbersome. 4.0 introduces the ability to easily enable call recordings and track all call recordings in CDRs. Each CDR points to the call recording for easy lookup and retrieval. The recordings themselves can be placed on an external storage engine, such as Amazon S3, making it easier for people to manage the recordings.
How does it impact you:
You must bring your own storage engine and configure this to use it.
Multiple call recordings per call
What’s New:
Sometimes, on a long call, users wish to start and stop recordings mid-call but still have a single file result at the end of the call. In addition, calls which are transferred between callers are expected to remain in a single file. We have upgraded the system to support storing multiple parts of a call in a single file.
How does it impact you:
There is no preparation required to take advantage of this improvement.
CDRs
Added interaction ids and Listed CDRs, now shown as calls
What’s New:
Kazoo has always delivered multiple records for every call because it creates a CDR for each leg of a call. This can be challenging to program for if you are a developer, and can confuse users. It also makes it challenging to understand what happened during a call. We are introducing a concept called “Interaction IDs” to solve this. Interaction IDs allow you to get a unified view of all calls in the system. You can drill down to the previously available detail about each call leg once you’ve located the interaction ID (call) of interest.
How does it impact you:
There is no preparation required to take advantage of this improvement.
RabbitMQ Enhancements
Clustering Rabbit over the local LAN
What’s New:
Kazoo now supports the initial attempt at redundant RabbitMQ clusters (on the local LAN). This solves the last remaining hurdle of full redundancy within a single zone. Using RabbitMQ’s built-in clustering, clusters can be configured to be aware of multiple AMQP nodes and Rabbit itself can use it’s built-in clustering to improve redundancy.
How does it impact you:
This solves the last full redundancy requirement for the stack. Enabling this and properly configuring all other components means the system is fully redundant in every level of the stack and should failover automatically. More testing is needed here, so please feel free to play with this.
Kamailio
TLS / SRTP
What’s New:
We are now configuring TLS and SRTP support out of the box. We will soon also utilize “letsencrypt.com” to auto-generate certificates. This will help with configuring and utilizing SRTP encryption and WebRTC services.
How does it impact you:
There is no preparation required to take advantage of this improvement.
HTTP / HTTPS
No more options pings on API calls (Preflight)
What’s New:
As typically deployed, the web UI communicates to a Kazoo API server. The two URLs to do this are not necessarily on the same domain. This causes browsers to send an OPTIONS pre-flight request which significantly slows down the UI. Through new deployment procedures, we are eliminating these extraneous requests.
How does it impact you:
There is no preparation required to take advantage of this improvement.
Tasks API
General
What’s New:
A major feature enhancement was added to 4.0 which allows for generic background processes to be queued across a cluster. This feature represents a framework for adding background jobs to a queue and then popping them off, processing them and returning the results. Individual APIs can now be built that take advantage of this queueing mechanism.
How does it impact you:
There is no preparation required to take advantage of this improvement.
Jonny5 / Ecallmgr / Call Rating & Limits
Community contribution to disconnect calls when the credit is exhausted
What’s New:
Kazoo’s ecallmgr can now be configured to continuously validate that active calls consuming per minute credits still have funds available. If the funds are exhausted the channel will be killed in the middle of the call.
How does it impact you:
By default per minute usage is debited from the account at the termination of a call and a long call could result in a negative balance. By checking proactively – this is avoided. This feature is community supported only (not supported by 2600Hz).
Carrier Management
Bandwidth 2 Carrier Edition
What’s New:
Bandwidth.com has stopped using the old API subsystem on my.bandwidth.com, which we used to rely on. This required re-coding the bandwidth.com module which allows for searching and purchasing of phone numbers. The new bandwidth API for purchasing and searching for numbers provides better service from the customer’s perspective and allows us to continue supporting this provider.
How does it impact you:
You’ll need to reconfigure the wnm_bandwidth module to use wnm_bandwidth2 on the database backend and enter new credentials / API keys for the new bandwidth.com dashboard.
Voip Innovations Carrier Edition
What’s New:
2600Hz has added backend support for VoIP Innovations as a carrier module. This allows Kazoo Administrators to search, buy numbers, configure E911 and manage settings for numbers on a VoIP Innovations wholesale account, through Kazoo.
How does it impact you:
You’ll need to reconfigure the wnm_bandwidth module to use wnm_voipinnovations on the database backend and enter new credentials / API keys for the new bandwidth.com dashboard.
APIs
Voicemail enhancements
What’s New:
Sometimes a customer’s voicemail box fills up erroneously, usually because it is misconfigured (but sometimes because the voicemail to email application is having an issue sending emails, in which case the voicemails are retained). This requires clearing out the voicemails in the mailbox later, which is tedious. This enhancement introduces the ability to clear a voicemail box all at once.
How does it impact you:
There is no preparation required to take advantage of this improvement.
HTTPS for all
What’s New:
We are now including HTTPS support as the default strategy for all interactions with Kazoo. Soon, we will sunset support for raw unencrypted HTTP requests.
How does it impact you:
You should switch your application to use HTTPS if you are calling our APIs directly.
WARNING: we have changed a lot in the numbers API
What’s New:
The phone numbers API was updated heavily. A lot of work was put into getting the new number manager to “speak” the same API as the old API used to. However, it’s possible 100% backward compatibility has not been obtained. Mishaps may occur.
How does it impact you:
If you have done your own integrations to our number manager, you should test APIs for phone numbers to be sure they work as expected. Most of the changes are in the DB storage mechanism and should be transparent but this will not be code we can roll back once it’s out, so please proceed with caution.
BUG FIXES
Callflows
On-net CDR fixes (loopback)
What’s New:
Kazoo now properly logs CDRs when calling between accounts on the same system. Previously, the CDR would only be in a single account.
How does it impact you:
There is no preparation required to take advantage of this improvement.
On-net authz fixes (loopback)
What’s New:
Kazoo now properly authorizes calls when calling between accounts. This allows billing to work properly when calling between accounts on the same cluster.
How does it impact you:
There is no preparation required to take advantage of this improvement.
On-net T.38 fixes
What’s New:
Kazoo now properly handles fax calls between two T.38 devices on the same cluster.
How does it impact you:
There is no preparation required to take advantage of this improvement.
On-hold music fixes
What’s New:
You can now transfer calls to your clients/across accounts and the correct on hold music plays for the right account when calling between accounts.
How does it impact you:
There is no preparation required to take advantage of this improvement.
Kamailio
Resolve contact string at Kamailio reducing the load on ecallmanager.
What’s New:
Previously, every phone which made a call would be asked for the phone’s username/password. This occurred on each call. The process of doing this can take 50-150ms in some cases, which slows down call setup times. In addition, the backend systems would have to re-check the password for a phone even though we already know the phone’s IP and port and know it’s authorized. Now, we don’t challenge already registered phones if the INVITE comes from the same IP and port as a previous call. This improves performance on things like park/transfer. This also reduces all call setup time and reduces post dial delay.
How does it impact you:
There is no preparation required to take advantage of this improvement.
WebRTC Support
What’s New:
We now properly lookup and address WebRTC clients when calling inbound. This will help ensure WebRTC connectivity when calling to WebRTC users.
How does it impact you:
There is no preparation required to take advantage of this improvement.
General
Forgot password update–now sends a link to actually reset password
What’s New:
The forgot password button previously reset a password on the fly and emailed the new password. Malicious users could submit false password requests that required no confirmation. The system has been updated to work more like other systems where an email with a token/link is sent first to confirm the user really requested the password reset, and then the password is reset once the link is clicked on.
How does it impact you:
You’ll need to make sure the new forgot password templates are updated.
Konami
Konami code for audio levels-adjust your speaker volume or mic sensitivity
What’s New:
The older version of Konami received an update that allows keypresses to adjust the volume level or mic sensitivity when mid-call.
How does it impact you:
You’ll need to be running Konami.
APIs
Blacklists
What’s New:
We have had blacklists for a while, but they have not had a way to handle Anonymous Caller IDs on inbound phone calls. Now the blacklist app will not attempt to try to normalize an anonymous caller ID, which means that blacklisting “all zeros” (anonymous caller ID) will now work correctly.
How does it impact you:
If you are using the blacklist APIs, you can now block anonymous caller IDs.
Validate type of document being posted
What’s New:
Previously, an issue existed where the API would not validate the pvt_type you submitted against the existing document. This means that if you were trying to update a voicemail box via the API, but accidentally passed in the document ID of something else (like the account or user), the system would happily accept the update – and break the corresponding document/feature you incorrectly specified. While this error is caused by the user/developer, Kazoo has been upgraded to be stricter about enforcing the pvt_type stays the same so that this error can’t be made accidentally.
How does it impact you:
If you are using our APIs from your own scripts, you should be aware of this validation in case you somehow cause an error by trying to change the pvt_type (although you should never do that anyway, so it just means you should fix your code).
