[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:39:38 dist-test-slave-v0k1 krb5kdc[18378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:39:38 dist-test-slave-v0k1 krb5kdc[18378](info): set up 2 sockets
May 06 15:39:38 dist-test-slave-v0k1 krb5kdc[18378](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20240506 15:39:39.726737 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:39:39 dist-test-slave-v0k1 krb5kdc[18378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:39:39.855530 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:33043
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:33043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:39.936156 18391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:39.936362 18391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:39.936383 18391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:39.937647 18391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:39:39.937693 18391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:39.937707 18391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:39.937719 18391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:39:39.937731 18391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:39:39.939332 18391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:33043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:33043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:39.939529 18391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:39.939688 18391 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:39.944789 18399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:39.945312 18396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:39.946183 18397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:39.946935 18391 server_base.cc:1034] running on GCE node
I20240506 15:39:39.947229 18391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:39.947547 18391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:39.948704 18391 hybrid_clock.cc:648] HybridClock initialized: now 1715009979948680 us; error 43 us; skew 500 ppm
May 06 15:39:39 dist-test-slave-v0k1 krb5kdc[18378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009979, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:39.951530 18391 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:39:39.951947 18391 webserver.cc:460] Webserver started at http://127.17.240.254:39133/ using document root <none> and password file <none>
I20240506 15:39:39.952127 18391 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:39.952164 18391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:39.952252 18391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:39.953233 18391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "02636d7a87e742a5bd19c5097ade9376"
format_stamp: "Formatted at 2024-05-06 15:39:39 on dist-test-slave-v0k1"
server_key: "3030ab70c0a8d6b39c95b240ecb64209"
server_key_iv: "62691c52845da81a37644fbc841b7dae"
server_key_version: "encryptionkey@0"
I20240506 15:39:39.953521 18391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "02636d7a87e742a5bd19c5097ade9376"
format_stamp: "Formatted at 2024-05-06 15:39:39 on dist-test-slave-v0k1"
server_key: "3030ab70c0a8d6b39c95b240ecb64209"
server_key_iv: "62691c52845da81a37644fbc841b7dae"
server_key_version: "encryptionkey@0"
I20240506 15:39:39.954742 18391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240506 15:39:39.955539 18406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:39.955744 18391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240506 15:39:39.955828 18391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "02636d7a87e742a5bd19c5097ade9376"
format_stamp: "Formatted at 2024-05-06 15:39:39 on dist-test-slave-v0k1"
server_key: "3030ab70c0a8d6b39c95b240ecb64209"
server_key_iv: "62691c52845da81a37644fbc841b7dae"
server_key_version: "encryptionkey@0"
I20240506 15:39:39.955886 18391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:40.004079 18391 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:40.006114 18391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:40.006219 18391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:40.010975 18391 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:33043
I20240506 15:39:40.011077 18458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:33043 every 8 connection(s)
I20240506 15:39:40.011485 18391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:39:40.012377 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:40.014425 18459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376: Bootstrap starting.
I20240506 15:39:40.014976 18459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:40.015244 18459 log.cc:824] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:40.015890 18459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376: No bootstrap required, opened a new log
I20240506 15:39:40.017277 18459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02636d7a87e742a5bd19c5097ade9376" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33043 } }
I20240506 15:39:40.017398 18459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:40.017416 18459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02636d7a87e742a5bd19c5097ade9376, State: Initialized, Role: FOLLOWER
I20240506 15:39:40.017526 18459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02636d7a87e742a5bd19c5097ade9376" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33043 } }
I20240506 15:39:40.017598 18459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:39:40.017621 18459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:39:40.017647 18459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:40.018173 18459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02636d7a87e742a5bd19c5097ade9376" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33043 } }
I20240506 15:39:40.018262 18459 leader_election.cc:302] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02636d7a87e742a5bd19c5097ade9376; no voters:
I20240506 15:39:40.018404 18459 leader_election.cc:288] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:39:40.018622 18462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:40.018638 18459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:39:40.018779 18462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [term 1 LEADER]: Becoming Leader. State: Replica: 02636d7a87e742a5bd19c5097ade9376, State: Running, Role: LEADER
I20240506 15:39:40.018867 18462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02636d7a87e742a5bd19c5097ade9376" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33043 } }
I20240506 15:39:40.019271 18464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02636d7a87e742a5bd19c5097ade9376. Latest consensus state: current_term: 1 leader_uuid: "02636d7a87e742a5bd19c5097ade9376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02636d7a87e742a5bd19c5097ade9376" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33043 } } }
I20240506 15:39:40.019249 18463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02636d7a87e742a5bd19c5097ade9376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02636d7a87e742a5bd19c5097ade9376" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33043 } } }
I20240506 15:39:40.019398 18464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:40.019402 18463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376 [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:40.019644 18471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:39:40.020092 18471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:39:40.020771 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 18391
I20240506 15:39:40.020879 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:39:40.021406 18471 catalog_manager.cc:1347] Generated new cluster ID: e4b32c19d6534754b421f581979c26e3
I20240506 15:39:40.021449 18471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:39:40.021574 18371 external_mini_cluster.cc:1428] Setting key 1a1a815aea82fc99b6bf986ac69c6823
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:40.035884 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:40.022236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.17.240.254:33043)
0506 15:39:40.024318 (+ 2082us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:40.024329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:40.024348 (+ 19us) server_negotiation.cc:407] Connection header received
0506 15:39:40.024439 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:40.024443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:40.024498 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:40.024605 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:40.026252 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.027467 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:40.030580 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.030826 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:40.032084 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:40.032092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:40.032092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:40.032110 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:40.034134 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.034578 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.034581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.034581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.034632 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.034815 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.034815 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.034816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.035060 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:40.035192 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:40.035431 (+ 239us) server_negotiation.cc:299] Negotiation successful
0506 15:39:40.035555 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2042,"thread_start_us":60,"threads_started":1}
I20240506 15:39:40.063781 18471 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:39:40.064352 18471 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:39:40.069644 18471 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 02636d7a87e742a5bd19c5097ade9376: Generated new TSK 0
I20240506 15:39:40.069837 18471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:39:40.098466 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33043
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:40.182250 18485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:40.182487 18485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:40.182512 18485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:40.183841 18485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:40.183881 18485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:40.183907 18485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:39:40.185626 18485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:40.185926 18485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:40.186153 18485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:40.191143 18493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:40.191686 18491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:40.192356 18490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:40.193110 18485 server_base.cc:1034] running on GCE node
I20240506 15:39:40.193300 18485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:40.193519 18485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:40.194689 18485 hybrid_clock.cc:648] HybridClock initialized: now 1715009980194671 us; error 35 us; skew 500 ppm
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009980, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:40.197170 18485 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:39:40.197569 18485 webserver.cc:460] Webserver started at http://127.17.240.193:42605/ using document root <none> and password file <none>
I20240506 15:39:40.197726 18485 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:40.197762 18485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:40.197844 18485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:40.198830 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "08577847aa574019a3fffc7987813849"
format_stamp: "Formatted at 2024-05-06 15:39:40 on dist-test-slave-v0k1"
server_key: "8fa0c1b2aa4bef0acd9a68a31d0509f3"
server_key_iv: "b840ed9ee80fb25a7f5f13bc19635bab"
server_key_version: "encryptionkey@0"
I20240506 15:39:40.199148 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "08577847aa574019a3fffc7987813849"
format_stamp: "Formatted at 2024-05-06 15:39:40 on dist-test-slave-v0k1"
server_key: "8fa0c1b2aa4bef0acd9a68a31d0509f3"
server_key_iv: "b840ed9ee80fb25a7f5f13bc19635bab"
server_key_version: "encryptionkey@0"
I20240506 15:39:40.200309 18485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:39:40.201009 18500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:40.201179 18485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:39:40.201258 18485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "08577847aa574019a3fffc7987813849"
format_stamp: "Formatted at 2024-05-06 15:39:40 on dist-test-slave-v0k1"
server_key: "8fa0c1b2aa4bef0acd9a68a31d0509f3"
server_key_iv: "b840ed9ee80fb25a7f5f13bc19635bab"
server_key_version: "encryptionkey@0"
I20240506 15:39:40.201318 18485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:40.230928 18485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:40.233379 18485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:40.233517 18485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:40.233738 18485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:40.234048 18485 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:40.234082 18485 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:40.234117 18485 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:40.234148 18485 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:40.239738 18485 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:41139
I20240506 15:39:40.239799 18613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:41139 every 8 connection(s)
I20240506 15:39:40.240072 18485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:39:40.243255 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 18485
I20240506 15:39:40.243356 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:39:40.243513 18371 external_mini_cluster.cc:1428] Setting key a58aeb988061c520e7b04289372f23d9
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009980, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:40.249100 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:40.240686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:56683 (local address 127.17.240.254:33043)
0506 15:39:40.240732 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:40.240733 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:40.240939 (+ 206us) server_negotiation.cc:407] Connection header received
0506 15:39:40.241458 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:40.241459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:40.241482 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:40.241547 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:40.242637 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.243175 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:40.243882 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.244080 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:40.246249 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:40.246254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:40.246254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:40.246263 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:40.247893 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.248292 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.248293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.248293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.248316 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.248484 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.248485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.248485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.248590 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:40.248648 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:40.248891 (+ 243us) server_negotiation.cc:299] Negotiation successful
0506 15:39:40.248993 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240506 15:39:40.249228 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:40.240678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:33043 (local address 127.17.240.193:56683)
0506 15:39:40.240825 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:40.240855 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:40.241335 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:40.241605 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:40.241609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:40.241911 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:40.242502 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:40.242506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.243286 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:40.243287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:40.243722 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:40.243724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.243869 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:40.244511 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:40.244528 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:40.246147 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:40.247997 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:40.248003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:40.248006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:40.248221 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:40.248368 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:40.248369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:40.248369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:40.248434 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:40.248641 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:40.248646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:40.248770 (+ 124us) client_negotiation.cc:769] Sending connection context
0506 15:39:40.248868 (+ 98us) client_negotiation.cc:240] Negotiation successful
0506 15:39:40.249028 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20240506 15:39:40.249747 18614 heartbeater.cc:347] Connected to a master server at 127.17.240.254:33043
I20240506 15:39:40.249838 18614 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:40.250046 18614 heartbeater.cc:511] Master 127.17.240.254:33043 requested a full tablet report, sending...
I20240506 15:39:40.250654 18423 ts_manager.cc:194] Registered new tserver with Master: 08577847aa574019a3fffc7987813849 (127.17.240.193:41139)
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
I20240506 15:39:40.251506 18423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:56683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:39:40.270855 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33043
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:40.350353 18619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:40.350564 18619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:40.350587 18619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:40.351907 18619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:40.351951 18619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:40.351976 18619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:39:40.353464 18619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:40.353653 18619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:40.353804 18619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:40.358867 18624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:40.359524 18625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:40.360183 18627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:40.360858 18619 server_base.cc:1034] running on GCE node
I20240506 15:39:40.361059 18619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:40.361266 18619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:40.362426 18619 hybrid_clock.cc:648] HybridClock initialized: now 1715009980362407 us; error 30 us; skew 500 ppm
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009980, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:40.365002 18619 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:39:40.365368 18619 webserver.cc:460] Webserver started at http://127.17.240.194:34125/ using document root <none> and password file <none>
I20240506 15:39:40.365522 18619 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:40.365558 18619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:40.365641 18619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:40.366573 18619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "e3d32422fc6047fe8692302fe86de031"
format_stamp: "Formatted at 2024-05-06 15:39:40 on dist-test-slave-v0k1"
server_key: "9efabee26463d15fe2fdd7440d61a822"
server_key_iv: "8d5fafcae15f309e9efc19d23f716a9a"
server_key_version: "encryptionkey@0"
I20240506 15:39:40.366832 18619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "e3d32422fc6047fe8692302fe86de031"
format_stamp: "Formatted at 2024-05-06 15:39:40 on dist-test-slave-v0k1"
server_key: "9efabee26463d15fe2fdd7440d61a822"
server_key_iv: "8d5fafcae15f309e9efc19d23f716a9a"
server_key_version: "encryptionkey@0"
I20240506 15:39:40.368018 18619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:39:40.368676 18634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:40.368854 18619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:39:40.368932 18619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "e3d32422fc6047fe8692302fe86de031"
format_stamp: "Formatted at 2024-05-06 15:39:40 on dist-test-slave-v0k1"
server_key: "9efabee26463d15fe2fdd7440d61a822"
server_key_iv: "8d5fafcae15f309e9efc19d23f716a9a"
server_key_version: "encryptionkey@0"
I20240506 15:39:40.369011 18619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:40.385907 18619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:40.387778 18619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:40.387877 18619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:40.388104 18619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:40.388397 18619 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:40.388429 18619 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:40.388482 18619 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:40.388509 18619 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:40.394065 18619 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:43001
I20240506 15:39:40.394104 18747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:43001 every 8 connection(s)
I20240506 15:39:40.394392 18619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:39:40.395190 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 18619
I20240506 15:39:40.395267 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:39:40.395412 18371 external_mini_cluster.cc:1428] Setting key b4d094c84e49fb75c8d7fd6e274b8208
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009980, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:40.403414 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:40.395001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:43259 (local address 127.17.240.254:33043)
0506 15:39:40.395115 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:40.395117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:40.395333 (+ 216us) server_negotiation.cc:407] Connection header received
0506 15:39:40.395922 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:40.395924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:40.395937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:40.395988 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:40.397444 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.397963 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:40.398636 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.398851 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:40.400849 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:40.400854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:40.400854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:40.400860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:40.402272 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.402618 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.402619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.402620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.402642 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.402796 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.402797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.402797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.402886 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:40.402944 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:40.403227 (+ 283us) server_negotiation.cc:299] Negotiation successful
0506 15:39:40.403318 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240506 15:39:40.403532 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:40.394973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:33043 (local address 127.17.240.194:43259)
0506 15:39:40.395217 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:40.395245 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:40.395825 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:40.396048 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:40.396052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:40.396534 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:40.397348 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:40.397355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.398046 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:40.398047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:40.398547 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:40.398549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.398670 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:40.399183 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:40.399196 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:40.400760 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:40.402346 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:40.402351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:40.402353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:40.402547 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:40.402692 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:40.402692 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:40.402693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:40.402747 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:40.402938 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:40.402942 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:40.403090 (+ 148us) client_negotiation.cc:769] Sending connection context
0506 15:39:40.403191 (+ 101us) client_negotiation.cc:240] Negotiation successful
0506 15:39:40.403333 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
I20240506 15:39:40.404028 18748 heartbeater.cc:347] Connected to a master server at 127.17.240.254:33043
I20240506 15:39:40.404232 18748 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:40.404446 18748 heartbeater.cc:511] Master 127.17.240.254:33043 requested a full tablet report, sending...
I20240506 15:39:40.404779 18423 ts_manager.cc:194] Registered new tserver with Master: e3d32422fc6047fe8692302fe86de031 (127.17.240.194:43001)
I20240506 15:39:40.405169 18423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:43259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:39:40.423368 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33043
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:40.502278 18753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:40.502471 18753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:40.502494 18753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:40.503839 18753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:40.503881 18753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:40.503906 18753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:39:40.505457 18753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:40.505645 18753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:40.505800 18753 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:40.510720 18758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:40.511173 18759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:40.511821 18761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:40.512581 18753 server_base.cc:1034] running on GCE node
I20240506 15:39:40.512730 18753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:40.512902 18753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:40.514035 18753 hybrid_clock.cc:648] HybridClock initialized: now 1715009980514024 us; error 26 us; skew 500 ppm
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009980, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:40.516443 18753 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:39:40.516851 18753 webserver.cc:460] Webserver started at http://127.17.240.195:46337/ using document root <none> and password file <none>
I20240506 15:39:40.517032 18753 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:40.517088 18753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:40.517225 18753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:40.518354 18753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "fbf7750cdffe4a5fb86de6b9c2f45622"
format_stamp: "Formatted at 2024-05-06 15:39:40 on dist-test-slave-v0k1"
server_key: "9230dde0227f1eb718cd4a924e994d9a"
server_key_iv: "80a2de4361a5325d4260242a03aef03e"
server_key_version: "encryptionkey@0"
I20240506 15:39:40.518667 18753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "fbf7750cdffe4a5fb86de6b9c2f45622"
format_stamp: "Formatted at 2024-05-06 15:39:40 on dist-test-slave-v0k1"
server_key: "9230dde0227f1eb718cd4a924e994d9a"
server_key_iv: "80a2de4361a5325d4260242a03aef03e"
server_key_version: "encryptionkey@0"
I20240506 15:39:40.519913 18753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:39:40.520658 18768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:40.520799 18753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:39:40.520870 18753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "fbf7750cdffe4a5fb86de6b9c2f45622"
format_stamp: "Formatted at 2024-05-06 15:39:40 on dist-test-slave-v0k1"
server_key: "9230dde0227f1eb718cd4a924e994d9a"
server_key_iv: "80a2de4361a5325d4260242a03aef03e"
server_key_version: "encryptionkey@0"
I20240506 15:39:40.520931 18753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:40.530794 18753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:40.532490 18753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:40.532590 18753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:40.532788 18753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:40.533066 18753 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:40.533099 18753 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:40.533131 18753 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:40.533166 18753 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:40.538748 18753 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:34399
I20240506 15:39:40.538795 18881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:34399 every 8 connection(s)
I20240506 15:39:40.539110 18753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009980, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:40.546576 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:40.539771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:52795 (local address 127.17.240.254:33043)
0506 15:39:40.539830 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:40.539831 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:40.539989 (+ 158us) server_negotiation.cc:407] Connection header received
0506 15:39:40.540438 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:40.540439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:40.540447 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:40.540512 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:40.541483 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.541998 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:40.542486 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.542588 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:40.544442 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:40.544446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:40.544446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:40.544452 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:40.545626 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.545946 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.545946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.545947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.545962 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.546116 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.546117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.546117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.546203 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:40.546247 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:40.546439 (+ 192us) server_negotiation.cc:299] Negotiation successful
0506 15:39:40.546507 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:39:40.546686 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:40.539717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:33043 (local address 127.17.240.195:52795)
0506 15:39:40.539889 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:40.539918 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:40.540359 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:40.540565 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:40.540569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:40.540857 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:40.541391 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:40.541396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.542062 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:40.542063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:40.542430 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:40.542431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.542530 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:40.543023 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:40.543042 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:40.544352 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:40.545697 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:40.545701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:40.545703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:40.545893 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:40.546021 (+ 128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:40.546021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:40.546022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:40.546074 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:40.546241 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:40.546244 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:40.546343 (+ 99us) client_negotiation.cc:769] Sending connection context
0506 15:39:40.546425 (+ 82us) client_negotiation.cc:240] Negotiation successful
0506 15:39:40.546532 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20240506 15:39:40.547147 18882 heartbeater.cc:347] Connected to a master server at 127.17.240.254:33043
I20240506 15:39:40.547231 18882 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:40.547391 18882 heartbeater.cc:511] Master 127.17.240.254:33043 requested a full tablet report, sending...
I20240506 15:39:40.547730 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 18753
I20240506 15:39:40.547773 18423 ts_manager.cc:194] Registered new tserver with Master: fbf7750cdffe4a5fb86de6b9c2f45622 (127.17.240.195:34399)
I20240506 15:39:40.547794 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:39:40.547935 18371 external_mini_cluster.cc:1428] Setting key b81af7ca0855349d32e760b864b367b0
I20240506 15:39:40.548216 18423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:52795
I20240506 15:39:40.548892 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009980, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009980, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:39:40.569792 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:40.562812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.17.240.193:41139)
0506 15:39:40.562996 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:40.563003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:40.563051 (+ 48us) server_negotiation.cc:407] Connection header received
0506 15:39:40.563086 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:40.563088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:40.563101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:40.563170 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:40.563812 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.564516 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:40.565090 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.565245 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:40.567251 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:40.567264 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:40.567264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:40.567271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:40.568896 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.569144 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.569146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.569146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.569172 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.569265 (+ 93us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.569265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.569266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.569369 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:40.569505 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:40.569552 (+ 47us) server_negotiation.cc:299] Negotiation successful
0506 15:39:40.569646 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
W20240506 15:39:40.570183 18528 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51658
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:39:40.591859 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:40.584908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.17.240.193:41139)
0506 15:39:40.584963 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:40.584965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:40.585056 (+ 91us) server_negotiation.cc:407] Connection header received
0506 15:39:40.585135 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:40.585136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:40.585150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:40.585193 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:40.585874 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.586361 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:40.587128 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:40.587295 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:40.589325 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:40.589331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:40.589331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:40.589338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:40.590735 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.591133 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.591134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.591135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.591157 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:40.591325 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:40.591325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:40.591326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:40.591396 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:40.591497 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:40.591633 (+ 136us) server_negotiation.cc:299] Negotiation successful
0506 15:39:40.591718 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240506 15:39:40.592509 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 18485
I20240506 15:39:40.597718 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 18619
I20240506 15:39:40.602958 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 18753
I20240506 15:39:40.607415 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 18391
2024-05-06T15:39:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1850 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18896](info): set up 2 sockets
May 06 15:39:40 dist-test-slave-v0k1 krb5kdc[18896](info): commencing operation
krb5kdc: starting...
W20240506 15:39:41.562670 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:39:41 dist-test-slave-v0k1 krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:39:41.682631 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:35727
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:35727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:41.761370 18909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:41.761560 18909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:41.761583 18909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:41.762811 18909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:39:41.762845 18909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:41.762858 18909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:41.762871 18909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:39:41.762881 18909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:39:41.764364 18909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:35727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:35727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:41.764529 18909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:41.764664 18909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:41.769225 18915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:41.769819 18914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:41.770534 18917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:41.771157 18909 server_base.cc:1034] running on GCE node
I20240506 15:39:41.771356 18909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:41.771584 18909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:41.772720 18909 hybrid_clock.cc:648] HybridClock initialized: now 1715009981772705 us; error 26 us; skew 500 ppm
May 06 15:39:41 dist-test-slave-v0k1 krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009981, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:41.775059 18909 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:39:41.775424 18909 webserver.cc:460] Webserver started at http://127.17.240.254:45467/ using document root <none> and password file <none>
I20240506 15:39:41.775578 18909 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:41.775614 18909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:41.775687 18909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:41.776623 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "cb58285b7a254474a007d4d04ef2e21b"
format_stamp: "Formatted at 2024-05-06 15:39:41 on dist-test-slave-v0k1"
server_key: "69c1eb9451072a5d2bc3fbf8b2cdd19d"
server_key_iv: "9fc25adc3d18e0a400315d768ad49b88"
server_key_version: "encryptionkey@0"
I20240506 15:39:41.776932 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "cb58285b7a254474a007d4d04ef2e21b"
format_stamp: "Formatted at 2024-05-06 15:39:41 on dist-test-slave-v0k1"
server_key: "69c1eb9451072a5d2bc3fbf8b2cdd19d"
server_key_iv: "9fc25adc3d18e0a400315d768ad49b88"
server_key_version: "encryptionkey@0"
I20240506 15:39:41.778100 18909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:39:41.778729 18924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:41.778923 18909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:41.779027 18909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "cb58285b7a254474a007d4d04ef2e21b"
format_stamp: "Formatted at 2024-05-06 15:39:41 on dist-test-slave-v0k1"
server_key: "69c1eb9451072a5d2bc3fbf8b2cdd19d"
server_key_iv: "9fc25adc3d18e0a400315d768ad49b88"
server_key_version: "encryptionkey@0"
I20240506 15:39:41.779090 18909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:41.793576 18909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:41.795557 18909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:41.795655 18909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:41.799364 18909 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:35727
I20240506 15:39:41.799396 18976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:35727 every 8 connection(s)
I20240506 15:39:41.799691 18909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:39:41.800470 18977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:41.802538 18977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b: Bootstrap starting.
I20240506 15:39:41.803062 18977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:41.803293 18977 log.cc:824] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:41.803921 18977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b: No bootstrap required, opened a new log
I20240506 15:39:41.805308 18977 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb58285b7a254474a007d4d04ef2e21b" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35727 } }
I20240506 15:39:41.805424 18977 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:41.805449 18977 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb58285b7a254474a007d4d04ef2e21b, State: Initialized, Role: FOLLOWER
I20240506 15:39:41.805542 18977 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb58285b7a254474a007d4d04ef2e21b" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35727 } }
I20240506 15:39:41.805596 18977 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:39:41.805635 18977 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:39:41.805683 18977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:41.806200 18977 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb58285b7a254474a007d4d04ef2e21b" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35727 } }
I20240506 15:39:41.806298 18977 leader_election.cc:302] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb58285b7a254474a007d4d04ef2e21b; no voters:
I20240506 15:39:41.806434 18977 leader_election.cc:288] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:39:41.806605 18980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:41.806661 18977 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:39:41.806746 18980 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [term 1 LEADER]: Becoming Leader. State: Replica: cb58285b7a254474a007d4d04ef2e21b, State: Running, Role: LEADER
I20240506 15:39:41.806833 18980 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb58285b7a254474a007d4d04ef2e21b" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35727 } }
I20240506 15:39:41.807209 18981 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb58285b7a254474a007d4d04ef2e21b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb58285b7a254474a007d4d04ef2e21b" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35727 } } }
I20240506 15:39:41.807231 18982 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb58285b7a254474a007d4d04ef2e21b. Latest consensus state: current_term: 1 leader_uuid: "cb58285b7a254474a007d4d04ef2e21b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb58285b7a254474a007d4d04ef2e21b" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35727 } } }
I20240506 15:39:41.807299 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 18909
I20240506 15:39:41.807309 18982 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:41.807377 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:39:41.807298 18981 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:41.807543 18371 external_mini_cluster.cc:1428] Setting key 43ebc1be7b2d007701e9d1d298e7fbb7
I20240506 15:39:41.807814 18989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:39:41.808712 18989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:39:41.810143 18989 catalog_manager.cc:1347] Generated new cluster ID: be118ed6fd864276b8e9515ea319a1c1
I20240506 15:39:41.810192 18989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:39:41 dist-test-slave-v0k1 krb5kdc[18896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:41.817229 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:41.808128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.17.240.254:35727)
0506 15:39:41.808277 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:41.808289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:41.808311 (+ 22us) server_negotiation.cc:407] Connection header received
0506 15:39:41.808478 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:41.808484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:41.808535 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:41.808681 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:41.809384 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:41.810348 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:41.811095 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:41.811331 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:41.813840 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:41.813846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:41.813846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:41.813864 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:41.815874 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:41.816250 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:41.816252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:41.816252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:41.816302 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:41.816434 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:41.816434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:41.816435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:41.816664 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:41.816781 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:41.816906 (+ 125us) server_negotiation.cc:299] Negotiation successful
0506 15:39:41.817034 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20240506 15:39:41.823354 18989 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:39:41.824095 18989 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:39:41.833504 18989 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cb58285b7a254474a007d4d04ef2e21b: Generated new TSK 0
I20240506 15:39:41.833705 18989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:39:41.862082 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35727
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:41.941440 19003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:41.941625 19003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:41.941659 19003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:41.942942 19003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:41.942986 19003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:41.943048 19003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:39:41.944638 19003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:41.944864 19003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:41.945036 19003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:41.949648 19008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:41.950299 19011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:41.950917 19009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:41.951607 19003 server_base.cc:1034] running on GCE node
I20240506 15:39:41.951787 19003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:41.952013 19003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:41.953166 19003 hybrid_clock.cc:648] HybridClock initialized: now 1715009981953143 us; error 34 us; skew 500 ppm
May 06 15:39:41 dist-test-slave-v0k1 krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009981, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:41.956084 19003 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:39:41.956593 19003 webserver.cc:460] Webserver started at http://127.17.240.193:46599/ using document root <none> and password file <none>
I20240506 15:39:41.956801 19003 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:41.956851 19003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:41.956959 19003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:41.958238 19003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "1f56cbb85109401090f4090c2b877ebd"
format_stamp: "Formatted at 2024-05-06 15:39:41 on dist-test-slave-v0k1"
server_key: "b67ff6e700c5e6e3f7b853f7641c8403"
server_key_iv: "91f735b2287694e9eb29ccdacea20b82"
server_key_version: "encryptionkey@0"
I20240506 15:39:41.958591 19003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "1f56cbb85109401090f4090c2b877ebd"
format_stamp: "Formatted at 2024-05-06 15:39:41 on dist-test-slave-v0k1"
server_key: "b67ff6e700c5e6e3f7b853f7641c8403"
server_key_iv: "91f735b2287694e9eb29ccdacea20b82"
server_key_version: "encryptionkey@0"
I20240506 15:39:41.960247 19003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:39:41.961036 19018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:41.961199 19003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:41.961279 19003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "1f56cbb85109401090f4090c2b877ebd"
format_stamp: "Formatted at 2024-05-06 15:39:41 on dist-test-slave-v0k1"
server_key: "b67ff6e700c5e6e3f7b853f7641c8403"
server_key_iv: "91f735b2287694e9eb29ccdacea20b82"
server_key_version: "encryptionkey@0"
I20240506 15:39:41.961347 19003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:41.969638 19003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:41.971383 19003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:41.971501 19003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:41.971714 19003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:41.971982 19003 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:41.972013 19003 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:41.972035 19003 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:41.972061 19003 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:41.977362 19003 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:46797
I20240506 15:39:41.977427 19131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:46797 every 8 connection(s)
I20240506 15:39:41.977674 19003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
May 06 15:39:41 dist-test-slave-v0k1 krb5kdc[18896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009981, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:41.985741 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:41.978354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:46171 (local address 127.17.240.254:35727)
0506 15:39:41.978410 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:41.978412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:41.978603 (+ 191us) server_negotiation.cc:407] Connection header received
0506 15:39:41.979100 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:41.979101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:41.979117 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:41.979174 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:41.979935 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:41.980428 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:41.981344 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:41.981488 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:41.983391 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:41.983395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:41.983395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:41.983402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:41.984774 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:41.985118 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:41.985118 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:41.985119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:41.985135 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:41.985262 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:41.985262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:41.985262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:41.985347 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:41.985392 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:41.985601 (+ 209us) server_negotiation.cc:299] Negotiation successful
0506 15:39:41.985670 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240506 15:39:41.985844 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:41.978317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:35727 (local address 127.17.240.193:46171)
0506 15:39:41.978490 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:41.978516 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:41.978957 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:41.979227 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:41.979230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:41.979533 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:41.979856 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:41.979860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:41.980493 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:41.980494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:41.981282 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:41.981283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:41.981389 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:41.981903 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:41.981918 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:41.983271 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:41.984847 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:41.984853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:41.984855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:41.985059 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:41.985176 (+ 117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:41.985176 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:41.985177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:41.985223 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:41.985386 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:41.985389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:41.985504 (+ 115us) client_negotiation.cc:769] Sending connection context
0506 15:39:41.985588 (+ 84us) client_negotiation.cc:240] Negotiation successful
0506 15:39:41.985689 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20240506 15:39:41.986335 19132 heartbeater.cc:347] Connected to a master server at 127.17.240.254:35727
I20240506 15:39:41.986447 19132 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:41.986654 19132 heartbeater.cc:511] Master 127.17.240.254:35727 requested a full tablet report, sending...
I20240506 15:39:41.986877 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 19003
I20240506 15:39:41.986982 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:39:41.987190 18371 external_mini_cluster.cc:1428] Setting key 9c55dccd2aefccc9dd9279dd4e36ae29
I20240506 15:39:41.987231 18941 ts_manager.cc:194] Registered new tserver with Master: 1f56cbb85109401090f4090c2b877ebd (127.17.240.193:46797)
I20240506 15:39:41.987963 18941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:46171
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:39:42.014117 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35727
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:42.094853 19137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:42.095116 19137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:42.095157 19137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:42.096529 19137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:42.096580 19137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:42.096606 19137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:39:42.098217 19137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:42.098438 19137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:42.098641 19137 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:42.103631 19143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:42.104054 19142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:42.104673 19145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:42.105360 19137 server_base.cc:1034] running on GCE node
I20240506 15:39:42.105566 19137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:42.105803 19137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:42.106971 19137 hybrid_clock.cc:648] HybridClock initialized: now 1715009982106960 us; error 35 us; skew 500 ppm
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009982, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:42.109635 19137 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:39:42.110072 19137 webserver.cc:460] Webserver started at http://127.17.240.194:34759/ using document root <none> and password file <none>
I20240506 15:39:42.110262 19137 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:42.110303 19137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:42.110407 19137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:42.111634 19137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "04f33738455c450089091002e8811b0b"
format_stamp: "Formatted at 2024-05-06 15:39:42 on dist-test-slave-v0k1"
server_key: "4ca39c8b9a76e0323b46c94298a4e154"
server_key_iv: "fe9708dbfdcff817ee5d0dd0b3e53f86"
server_key_version: "encryptionkey@0"
I20240506 15:39:42.111948 19137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "04f33738455c450089091002e8811b0b"
format_stamp: "Formatted at 2024-05-06 15:39:42 on dist-test-slave-v0k1"
server_key: "4ca39c8b9a76e0323b46c94298a4e154"
server_key_iv: "fe9708dbfdcff817ee5d0dd0b3e53f86"
server_key_version: "encryptionkey@0"
I20240506 15:39:42.113274 19137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:39:42.114079 19152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:42.114264 19137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:42.114351 19137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "04f33738455c450089091002e8811b0b"
format_stamp: "Formatted at 2024-05-06 15:39:42 on dist-test-slave-v0k1"
server_key: "4ca39c8b9a76e0323b46c94298a4e154"
server_key_iv: "fe9708dbfdcff817ee5d0dd0b3e53f86"
server_key_version: "encryptionkey@0"
I20240506 15:39:42.114406 19137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:42.130292 19137 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:42.132692 19137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:42.132814 19137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:42.133001 19137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:42.133278 19137 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:42.133311 19137 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:42.133333 19137 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:42.133347 19137 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:42.138629 19137 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:36647
I20240506 15:39:42.138684 19265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:36647 every 8 connection(s)
I20240506 15:39:42.139001 19137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[18896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009982, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:42.148458 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.139747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:33649 (local address 127.17.240.254:35727)
0506 15:39:42.139798 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.139800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.139963 (+ 163us) server_negotiation.cc:407] Connection header received
0506 15:39:42.140416 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.140417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.140426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.140492 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:42.141441 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.141992 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.143043 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.143207 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.145451 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:42.145456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:42.145457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:42.145464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:42.147071 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.147472 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.147473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.147473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.147497 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.147765 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.147766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.147766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.147870 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:42.147939 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.148302 (+ 363us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.148387 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240506 15:39:42.148545 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.139711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:35727 (local address 127.17.240.194:33649)
0506 15:39:42.139853 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:42.139885 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:42.140328 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:42.140637 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:42.140642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:42.140999 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:42.141305 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.141309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.142092 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.142093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:42.142897 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.142899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.143048 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.143621 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:42.143639 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:42.145232 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:42.147168 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:42.147174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:42.147177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:42.147400 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:42.147578 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:42.147579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:42.147579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:42.147656 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:42.148019 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:42.148024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:42.148150 (+ 126us) client_negotiation.cc:769] Sending connection context
0506 15:39:42.148258 (+ 108us) client_negotiation.cc:240] Negotiation successful
0506 15:39:42.148382 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20240506 15:39:42.149081 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 19137
I20240506 15:39:42.149161 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:39:42.149096 19266 heartbeater.cc:347] Connected to a master server at 127.17.240.254:35727
I20240506 15:39:42.149245 19266 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:42.149401 18371 external_mini_cluster.cc:1428] Setting key 6689b6a1b05cca18116ce368b28ecb7e
I20240506 15:39:42.149663 19266 heartbeater.cc:511] Master 127.17.240.254:35727 requested a full tablet report, sending...
I20240506 15:39:42.150228 18941 ts_manager.cc:194] Registered new tserver with Master: 04f33738455c450089091002e8811b0b (127.17.240.194:36647)
I20240506 15:39:42.150652 18941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:33649
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:39:42.175524 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35727
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:42.256435 19271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:42.256636 19271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:42.256668 19271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:42.258004 19271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:42.258056 19271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:42.258086 19271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:39:42.259750 19271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:42.259946 19271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:42.260138 19271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:42.265056 19276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:42.265551 19277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:42.266216 19279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:42.266860 19271 server_base.cc:1034] running on GCE node
I20240506 15:39:42.267072 19271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:42.267295 19271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:42.268425 19271 hybrid_clock.cc:648] HybridClock initialized: now 1715009982268413 us; error 28 us; skew 500 ppm
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009982, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:42.270834 19271 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:39:42.271261 19271 webserver.cc:460] Webserver started at http://127.17.240.195:34485/ using document root <none> and password file <none>
I20240506 15:39:42.271430 19271 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:42.271484 19271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:42.271610 19271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:42.272607 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "564345e07618495da6c03548f6758e6f"
format_stamp: "Formatted at 2024-05-06 15:39:42 on dist-test-slave-v0k1"
server_key: "9b8d0c64992e59db85af86dca72f4357"
server_key_iv: "14ac89fc7474f9f08333a17537e8b723"
server_key_version: "encryptionkey@0"
I20240506 15:39:42.272919 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "564345e07618495da6c03548f6758e6f"
format_stamp: "Formatted at 2024-05-06 15:39:42 on dist-test-slave-v0k1"
server_key: "9b8d0c64992e59db85af86dca72f4357"
server_key_iv: "14ac89fc7474f9f08333a17537e8b723"
server_key_version: "encryptionkey@0"
I20240506 15:39:42.274078 19271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:39:42.274721 19286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:42.274884 19271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:42.274958 19271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "564345e07618495da6c03548f6758e6f"
format_stamp: "Formatted at 2024-05-06 15:39:42 on dist-test-slave-v0k1"
server_key: "9b8d0c64992e59db85af86dca72f4357"
server_key_iv: "14ac89fc7474f9f08333a17537e8b723"
server_key_version: "encryptionkey@0"
I20240506 15:39:42.275034 19271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:42.306823 19271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:42.309170 19271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:42.309305 19271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:42.309509 19271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:42.309820 19271 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:42.309854 19271 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:42.309890 19271 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:42.309914 19271 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:42.315426 19271 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:41143
I20240506 15:39:42.315464 19399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:41143 every 8 connection(s)
I20240506 15:39:42.315778 19271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:39:42.320410 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 19271
I20240506 15:39:42.320490 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:39:42.320637 18371 external_mini_cluster.cc:1428] Setting key b1a7264eb30473f1af85acf68d05697d
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[18896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009982, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:42.324988 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.316447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:54455 (local address 127.17.240.254:35727)
0506 15:39:42.316491 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.316493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.316690 (+ 197us) server_negotiation.cc:407] Connection header received
0506 15:39:42.317162 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.317163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.317174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.317257 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:42.318258 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.318918 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.319855 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.319981 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.322204 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:42.322208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:42.322208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:42.322214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:42.323825 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.324207 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.324209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.324209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.324235 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.324391 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.324391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.324391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.324524 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:42.324580 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.324842 (+ 262us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.324920 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240506 15:39:42.325096 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.316434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:35727 (local address 127.17.240.195:54455)
0506 15:39:42.316578 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:42.316603 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:42.317045 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:42.317301 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:42.317305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:42.317728 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:42.318140 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.318146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.318990 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.318991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:42.319774 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.319776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.319890 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.320454 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:42.320472 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:42.322103 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:42.323905 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:42.323910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:42.323912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:42.324120 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:42.324297 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:42.324298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:42.324298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:42.324346 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:42.324577 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:42.324580 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:42.324702 (+ 122us) client_negotiation.cc:769] Sending connection context
0506 15:39:42.324797 (+ 95us) client_negotiation.cc:240] Negotiation successful
0506 15:39:42.324906 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":90,"threads_started":1}
I20240506 15:39:42.325553 19400 heartbeater.cc:347] Connected to a master server at 127.17.240.254:35727
I20240506 15:39:42.325650 19400 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:42.325851 19400 heartbeater.cc:511] Master 127.17.240.254:35727 requested a full tablet report, sending...
I20240506 15:39:42.326211 18941 ts_manager.cc:194] Registered new tserver with Master: 564345e07618495da6c03548f6758e6f (127.17.240.195:41143)
I20240506 15:39:42.326584 18941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:54455
I20240506 15:39:42.332302 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:39:42.339439 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.333818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.17.240.254:35727)
0506 15:39:42.333876 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.333879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.333979 (+ 100us) server_negotiation.cc:407] Connection header received
0506 15:39:42.334037 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.334038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.334049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.334110 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:42.334783 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.335315 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.335872 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.335977 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.336972 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:42.336975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:42.336976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:42.336981 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:42.338334 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.338723 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.338724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.338725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.338751 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.338894 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.338894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.338894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.339059 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:42.339145 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.339262 (+ 117us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.339340 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240506 15:39:42.341773 18941 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:39:42.343271 18941 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:39:42.348555 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.345280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:46797 (local address 127.0.0.1:59030)
0506 15:39:42.345373 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:42.345400 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:42.345469 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:42.345709 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:42.345712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:42.345716 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:42.345983 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.345986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.347422 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.347423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:42.348233 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.348234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.348289 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.348327 (+ 38us) client_negotiation.cc:769] Sending connection context
0506 15:39:42.348383 (+ 56us) client_negotiation.cc:240] Negotiation successful
0506 15:39:42.348424 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":40,"threads_started":1}
I20240506 15:39:42.348783 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.345476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:36647 (local address 127.0.0.1:50330)
0506 15:39:42.345681 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:42.345692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:42.345740 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:42.345998 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:42.345998 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:42.346001 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:42.346162 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.346164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.347302 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.347302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:42.348212 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.348214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.348291 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.348327 (+ 36us) client_negotiation.cc:769] Sending connection context
0506 15:39:42.348383 (+ 56us) client_negotiation.cc:240] Negotiation successful
0506 15:39:42.348424 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20240506 15:39:42.349233 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.345359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.17.240.193:46797)
0506 15:39:42.345510 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.345516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.345524 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:39:42.345556 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.345558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.345570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.346075 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:42.346153 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.347351 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.348522 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.349028 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.349079 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.349114 (+ 35us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.349154 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20240506 15:39:42.349299 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.345676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.17.240.194:36647)
0506 15:39:42.345836 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.345843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.345851 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:39:42.345884 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.345886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.345900 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.345974 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:42.346264 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.347238 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.348497 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.349120 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.349152 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.349180 (+ 28us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.349215 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20240506 15:39:42.349462 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.345824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.17.240.195:41143)
0506 15:39:42.345959 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.345968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.346153 (+ 185us) server_negotiation.cc:407] Connection header received
0506 15:39:42.346367 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.346369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.346475 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.346563 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:42.346837 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.347741 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.348912 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.349254 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.349287 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.349320 (+ 33us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.349365 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20240506 15:39:42.349674 19414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.345638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:41143 (local address 127.0.0.1:36874)
0506 15:39:42.346072 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:42.346082 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:42.346221 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:42.346564 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:42.346565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:42.346566 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:42.346758 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.346760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.347809 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.347810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:42.348533 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.348537 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.348611 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.348615 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:39:42.349566 (+ 951us) client_negotiation.cc:240] Negotiation successful
0506 15:39:42.349619 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":72,"threads_started":1}
I20240506 15:39:42.349812 19066 tablet_service.cc:1467] Processing CreateTablet for tablet d39a65533f1e47369504ac5ec8ba0273 (DEFAULT_TABLE table=test-table [id=36d737e772bf48a29eb9e40f6f1b0b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:42.350128 19066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39a65533f1e47369504ac5ec8ba0273. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:42.350225 19334 tablet_service.cc:1467] Processing CreateTablet for tablet d39a65533f1e47369504ac5ec8ba0273 (DEFAULT_TABLE table=test-table [id=36d737e772bf48a29eb9e40f6f1b0b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:42.350484 19334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39a65533f1e47369504ac5ec8ba0273. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:42.351322 19200 tablet_service.cc:1467] Processing CreateTablet for tablet d39a65533f1e47369504ac5ec8ba0273 (DEFAULT_TABLE table=test-table [id=36d737e772bf48a29eb9e40f6f1b0b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:42.351567 19200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39a65533f1e47369504ac5ec8ba0273. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:42.354180 19419 tablet_bootstrap.cc:492] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b: Bootstrap starting.
I20240506 15:39:42.354243 19417 tablet_bootstrap.cc:492] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f: Bootstrap starting.
I20240506 15:39:42.354709 19417 tablet_bootstrap.cc:654] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:42.354712 19419 tablet_bootstrap.cc:654] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:42.354944 19417 log.cc:824] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:42.354959 19419 log.cc:824] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:42.355367 19418 tablet_bootstrap.cc:492] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd: Bootstrap starting.
I20240506 15:39:42.355696 19417 tablet_bootstrap.cc:492] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f: No bootstrap required, opened a new log
I20240506 15:39:42.355758 19417 ts_tablet_manager.cc:1367] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:39:42.355767 19419 tablet_bootstrap.cc:492] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b: No bootstrap required, opened a new log
I20240506 15:39:42.355831 19419 ts_tablet_manager.cc:1367] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:39:42.356127 19418 tablet_bootstrap.cc:654] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:42.356418 19418 log.cc:824] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:42.357322 19418 tablet_bootstrap.cc:492] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd: No bootstrap required, opened a new log
I20240506 15:39:42.357395 19418 ts_tablet_manager.cc:1367] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:39:42.357722 19417 raft_consensus.cc:348] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f56cbb85109401090f4090c2b877ebd" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46797 } } peers { permanent_uuid: "04f33738455c450089091002e8811b0b" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36647 } } peers { permanent_uuid: "564345e07618495da6c03548f6758e6f" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41143 } }
I20240506 15:39:42.357838 19417 raft_consensus.cc:374] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:42.357862 19417 raft_consensus.cc:729] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564345e07618495da6c03548f6758e6f, State: Initialized, Role: FOLLOWER
I20240506 15:39:42.357998 19417 consensus_queue.cc:260] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f56cbb85109401090f4090c2b877ebd" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46797 } } peers { permanent_uuid: "04f33738455c450089091002e8811b0b" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36647 } } peers { permanent_uuid: "564345e07618495da6c03548f6758e6f" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41143 } }
I20240506 15:39:42.358235 19417 ts_tablet_manager.cc:1398] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 15:39:42.358335 19400 heartbeater.cc:503] Master 127.17.240.254:35727 was elected leader, sending a full tablet report...
I20240506 15:39:42.358413 19419 raft_consensus.cc:348] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f56cbb85109401090f4090c2b877ebd" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46797 } } peers { permanent_uuid: "04f33738455c450089091002e8811b0b" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36647 } } peers { permanent_uuid: "564345e07618495da6c03548f6758e6f" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41143 } }
I20240506 15:39:42.358556 19419 raft_consensus.cc:374] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:42.358582 19419 raft_consensus.cc:729] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f33738455c450089091002e8811b0b, State: Initialized, Role: FOLLOWER
I20240506 15:39:42.358713 19419 consensus_queue.cc:260] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f56cbb85109401090f4090c2b877ebd" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46797 } } peers { permanent_uuid: "04f33738455c450089091002e8811b0b" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36647 } } peers { permanent_uuid: "564345e07618495da6c03548f6758e6f" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41143 } }
I20240506 15:39:42.358949 19419 ts_tablet_manager.cc:1398] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240506 15:39:42.358985 19266 heartbeater.cc:503] Master 127.17.240.254:35727 was elected leader, sending a full tablet report...
I20240506 15:39:42.359880 19418 raft_consensus.cc:348] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f56cbb85109401090f4090c2b877ebd" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46797 } } peers { permanent_uuid: "04f33738455c450089091002e8811b0b" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36647 } } peers { permanent_uuid: "564345e07618495da6c03548f6758e6f" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41143 } }
I20240506 15:39:42.360013 19418 raft_consensus.cc:374] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:42.360033 19418 raft_consensus.cc:729] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f56cbb85109401090f4090c2b877ebd, State: Initialized, Role: FOLLOWER
I20240506 15:39:42.360103 19418 consensus_queue.cc:260] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f56cbb85109401090f4090c2b877ebd" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46797 } } peers { permanent_uuid: "04f33738455c450089091002e8811b0b" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36647 } } peers { permanent_uuid: "564345e07618495da6c03548f6758e6f" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41143 } }
I20240506 15:39:42.360275 19418 ts_tablet_manager.cc:1398] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:39:42.360320 19132 heartbeater.cc:503] Master 127.17.240.254:35727 was elected leader, sending a full tablet report...
W20240506 15:39:42.389549 19267 tablet.cc:2369] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:39:42.478338 19133 tablet.cc:2369] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:39:42.540854 19423 raft_consensus.cc:482] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:39:42.541000 19423 raft_consensus.cc:504] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f56cbb85109401090f4090c2b877ebd" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46797 } } peers { permanent_uuid: "04f33738455c450089091002e8811b0b" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36647 } } peers { permanent_uuid: "564345e07618495da6c03548f6758e6f" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41143 } }
I20240506 15:39:42.541327 19423 leader_election.cc:288] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f56cbb85109401090f4090c2b877ebd (127.17.240.193:46797), 04f33738455c450089091002e8811b0b (127.17.240.194:36647)
I20240506 15:39:42.544055 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.541500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:46797 (local address 127.17.240.195:36991)
0506 15:39:42.541575 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:42.541588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:42.541636 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:42.541824 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:42.541825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:42.541827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:42.542084 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.542086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.542959 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.542960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:42.543814 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.543816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.543895 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.543900 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:39:42.543929 (+ 29us) client_negotiation.cc:240] Negotiation successful
0506 15:39:42.543962 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20240506 15:39:42.544230 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.541639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:36647 (local address 127.17.240.195:40529)
0506 15:39:42.541847 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:42.541864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:42.541944 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:42.542072 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:42.542073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:42.542074 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:42.542329 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.542331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.543197 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.543198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:42.544003 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:42.544004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.544064 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.544079 (+ 15us) client_negotiation.cc:769] Sending connection context
0506 15:39:42.544112 (+ 33us) client_negotiation.cc:240] Negotiation successful
0506 15:39:42.544146 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":122,"threads_started":1}
I20240506 15:39:42.544442 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.541602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:36991 (local address 127.17.240.193:46797)
0506 15:39:42.541661 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.541663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.541677 (+ 14us) server_negotiation.cc:407] Connection header received
0506 15:39:42.541709 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.541710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.541720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.541796 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:42.542160 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.542832 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.543901 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.544314 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.544323 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.544344 (+ 21us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.544373 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240506 15:39:42.544591 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.541742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:40529 (local address 127.17.240.194:36647)
0506 15:39:42.541802 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.541804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.541939 (+ 135us) server_negotiation.cc:407] Connection header received
0506 15:39:42.541997 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.541997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.542004 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.542050 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:42.542392 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.543110 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.544072 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.544467 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.544473 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.544494 (+ 21us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.544524 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:39:42.544780 19086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39a65533f1e47369504ac5ec8ba0273" candidate_uuid: "564345e07618495da6c03548f6758e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f56cbb85109401090f4090c2b877ebd" is_pre_election: true
I20240506 15:39:42.544845 19220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39a65533f1e47369504ac5ec8ba0273" candidate_uuid: "564345e07618495da6c03548f6758e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f33738455c450089091002e8811b0b" is_pre_election: true
I20240506 15:39:42.544950 19220 raft_consensus.cc:2454] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 564345e07618495da6c03548f6758e6f in term 0.
I20240506 15:39:42.544951 19086 raft_consensus.cc:2454] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 564345e07618495da6c03548f6758e6f in term 0.
I20240506 15:39:42.545162 19290 leader_election.cc:302] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f56cbb85109401090f4090c2b877ebd, 564345e07618495da6c03548f6758e6f; no voters:
I20240506 15:39:42.545289 19423 raft_consensus.cc:2781] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:39:42.545347 19423 raft_consensus.cc:482] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:39:42.545372 19423 raft_consensus.cc:3037] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:42.546048 19423 raft_consensus.cc:504] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f56cbb85109401090f4090c2b877ebd" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46797 } } peers { permanent_uuid: "04f33738455c450089091002e8811b0b" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36647 } } peers { permanent_uuid: "564345e07618495da6c03548f6758e6f" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41143 } }
I20240506 15:39:42.546165 19423 leader_election.cc:288] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [CANDIDATE]: Term 1 election: Requested vote from peers 1f56cbb85109401090f4090c2b877ebd (127.17.240.193:46797), 04f33738455c450089091002e8811b0b (127.17.240.194:36647)
I20240506 15:39:42.546357 19086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39a65533f1e47369504ac5ec8ba0273" candidate_uuid: "564345e07618495da6c03548f6758e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f56cbb85109401090f4090c2b877ebd"
I20240506 15:39:42.546372 19220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39a65533f1e47369504ac5ec8ba0273" candidate_uuid: "564345e07618495da6c03548f6758e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f33738455c450089091002e8811b0b"
I20240506 15:39:42.546448 19086 raft_consensus.cc:3037] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:42.546447 19220 raft_consensus.cc:3037] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:42.547370 19086 raft_consensus.cc:2454] T d39a65533f1e47369504ac5ec8ba0273 P 1f56cbb85109401090f4090c2b877ebd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 564345e07618495da6c03548f6758e6f in term 1.
I20240506 15:39:42.547515 19220 raft_consensus.cc:2454] T d39a65533f1e47369504ac5ec8ba0273 P 04f33738455c450089091002e8811b0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 564345e07618495da6c03548f6758e6f in term 1.
I20240506 15:39:42.547576 19290 leader_election.cc:302] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f56cbb85109401090f4090c2b877ebd, 564345e07618495da6c03548f6758e6f; no voters:
I20240506 15:39:42.547683 19423 raft_consensus.cc:2781] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:42.547820 19423 raft_consensus.cc:686] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [term 1 LEADER]: Becoming Leader. State: Replica: 564345e07618495da6c03548f6758e6f, State: Running, Role: LEADER
I20240506 15:39:42.547921 19423 consensus_queue.cc:237] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f56cbb85109401090f4090c2b877ebd" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46797 } } peers { permanent_uuid: "04f33738455c450089091002e8811b0b" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36647 } } peers { permanent_uuid: "564345e07618495da6c03548f6758e6f" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41143 } }
I20240506 15:39:42.548691 18941 catalog_manager.cc:5484] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 564345e07618495da6c03548f6758e6f (127.17.240.195). New cstate: current_term: 1 leader_uuid: "564345e07618495da6c03548f6758e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f56cbb85109401090f4090c2b877ebd" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f33738455c450089091002e8811b0b" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "564345e07618495da6c03548f6758e6f" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41143 } health_report { overall_health: HEALTHY } } }
W20240506 15:39:42.566260 19401 tablet.cc:2369] T d39a65533f1e47369504ac5ec8ba0273 P 564345e07618495da6c03548f6758e6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[18896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:39:42.579871 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.572935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.17.240.193:46797)
0506 15:39:42.573036 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.573039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.573081 (+ 42us) server_negotiation.cc:407] Connection header received
0506 15:39:42.573165 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.573166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.573176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.573228 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:42.573938 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.574404 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.574933 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.575166 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.577270 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:42.577283 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:42.577283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:42.577290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:42.579094 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.579376 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.579378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.579379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.579409 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.579502 (+ 93us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.579503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.579503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.579593 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:42.579698 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.579728 (+ 30us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.579798 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[18896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:39:42.601192 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.594952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.17.240.193:46797)
0506 15:39:42.595063 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.595065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.595121 (+ 56us) server_negotiation.cc:407] Connection header received
0506 15:39:42.595188 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.595189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.595198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.595257 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:42.595896 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.596369 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.596920 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.597058 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.599000 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:42.599027 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:42.599027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:42.599034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:42.600363 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.600667 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.600668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.600669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.600684 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.600785 (+ 101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.600786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.600786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.600852 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:42.600917 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.601042 (+ 125us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.601115 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[18896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:39:42.622066 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:42.616112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59076 (local address 127.17.240.193:46797)
0506 15:39:42.616167 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:42.616169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:42.616242 (+ 73us) server_negotiation.cc:407] Connection header received
0506 15:39:42.616335 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:42.616337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:42.616345 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:42.616397 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:42.617075 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.617548 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:42.618079 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:42.618192 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:42.620112 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:42.620118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:42.620118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:42.620124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:42.621327 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.621602 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.621602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.621603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.621618 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:42.621711 (+ 93us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:42.621712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:42.621712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:42.621778 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:42.621841 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:42.621918 (+ 77us) server_negotiation.cc:299] Negotiation successful
0506 15:39:42.621978 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20240506 15:39:42.622560 19046 tablet_service.cc:2908] Rejecting scan request for tablet d39a65533f1e47369504ac5ec8ba0273: Uninitialized: clean time has not yet been initialized
I20240506 15:39:42.623287 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 19003
I20240506 15:39:42.628435 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 19137
I20240506 15:39:42.633072 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 19271
I20240506 15:39:42.637333 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 18909
2024-05-06T15:39:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2030 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[19441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[19441](info): set up 2 sockets
May 06 15:39:42 dist-test-slave-v0k1 krb5kdc[19441](info): commencing operation
krb5kdc: starting...
W20240506 15:39:43.682132 18371 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20240506 15:39:43.884223 18371 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.202s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20240506 15:39:44.202454 18371 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.318s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20240506 15:39:44.410423 18371 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.208s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/test-user.keytab.
W20240506 15:39:45.285583 18371 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.875s user 0.000s sys 0.002s
May 06 15:39:45 dist-test-slave-v0k1 krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:39:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
W20240506 15:39:46.445698 18371 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.17.240.254: real 1.140s user 0.000s sys 0.003s
I20240506 15:39:46.446097 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39559
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:39559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:46.528136 19454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:46.528321 19454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:46.528343 19454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:46.529671 19454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:39:46.529719 19454 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240506 15:39:46.529732 19454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:46.529748 19454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:46.529775 19454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:39:46.529790 19454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:39:46.531472 19454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:39559
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:46.531698 19454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:46.531898 19454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:46.536543 19462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:46.537235 19459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:46.537890 19460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:46.538650 19454 server_base.cc:1034] running on GCE node
I20240506 15:39:46.538892 19454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:46.539157 19454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:46.540289 19454 hybrid_clock.cc:648] HybridClock initialized: now 1715009986540270 us; error 32 us; skew 500 ppm
May 06 15:39:46 dist-test-slave-v0k1 krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009986, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:46.547087 19454 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:39:46.547533 19454 webserver.cc:460] Webserver started at http://127.17.240.254:46135/ using document root <none> and password file <none>
I20240506 15:39:46.547734 19454 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:46.547780 19454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:46.547901 19454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:46.548915 19454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "4a504882e9134d2c86036cf59d6e5a67"
format_stamp: "Formatted at 2024-05-06 15:39:46 on dist-test-slave-v0k1"
server_key: "6a919074fb71c21261c65f05a2217462"
server_key_iv: "8310d0a1527980618134f7c278c130ed"
server_key_version: "encryptionkey@0"
I20240506 15:39:46.549225 19454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "4a504882e9134d2c86036cf59d6e5a67"
format_stamp: "Formatted at 2024-05-06 15:39:46 on dist-test-slave-v0k1"
server_key: "6a919074fb71c21261c65f05a2217462"
server_key_iv: "8310d0a1527980618134f7c278c130ed"
server_key_version: "encryptionkey@0"
I20240506 15:39:46.550442 19454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:39:46.551146 19469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:46.551302 19454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:46.551381 19454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "4a504882e9134d2c86036cf59d6e5a67"
format_stamp: "Formatted at 2024-05-06 15:39:46 on dist-test-slave-v0k1"
server_key: "6a919074fb71c21261c65f05a2217462"
server_key_iv: "8310d0a1527980618134f7c278c130ed"
server_key_version: "encryptionkey@0"
I20240506 15:39:46.551462 19454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:46.566569 19454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:46.568687 19454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:46.568814 19454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:46.573387 19454 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:39559
I20240506 15:39:46.573437 19531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:39559 every 8 connection(s)
I20240506 15:39:46.573762 19454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:39:46.574340 19532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:46.576510 19532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67: Bootstrap starting.
I20240506 15:39:46.577081 19532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:46.577342 19532 log.cc:824] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:46.577980 19532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67: No bootstrap required, opened a new log
I20240506 15:39:46.579438 19532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a504882e9134d2c86036cf59d6e5a67" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39559 } }
I20240506 15:39:46.579567 19532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:46.579586 19532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a504882e9134d2c86036cf59d6e5a67, State: Initialized, Role: FOLLOWER
I20240506 15:39:46.579699 19532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a504882e9134d2c86036cf59d6e5a67" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39559 } }
I20240506 15:39:46.579767 19532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:39:46.579789 19532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:39:46.579834 19532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:46.580709 19532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a504882e9134d2c86036cf59d6e5a67" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39559 } }
I20240506 15:39:46.580806 19532 leader_election.cc:302] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a504882e9134d2c86036cf59d6e5a67; no voters:
I20240506 15:39:46.580976 19532 leader_election.cc:288] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:39:46.581027 19535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:46.581058 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 19454
I20240506 15:39:46.581146 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:39:46.581197 19535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [term 1 LEADER]: Becoming Leader. State: Replica: 4a504882e9134d2c86036cf59d6e5a67, State: Running, Role: LEADER
I20240506 15:39:46.581236 19532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:39:46.581269 19535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a504882e9134d2c86036cf59d6e5a67" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39559 } }
I20240506 15:39:46.581321 18371 external_mini_cluster.cc:1428] Setting key 40bbba5ed15be8384bec752f880b5e48
I20240506 15:39:46.581542 19535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a504882e9134d2c86036cf59d6e5a67. Latest consensus state: current_term: 1 leader_uuid: "4a504882e9134d2c86036cf59d6e5a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a504882e9134d2c86036cf59d6e5a67" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39559 } } }
I20240506 15:39:46.581691 19535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:46.581868 19536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a504882e9134d2c86036cf59d6e5a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a504882e9134d2c86036cf59d6e5a67" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39559 } } }
I20240506 15:39:46.582018 19536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67 [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:46.582172 19543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:39:46.582743 19543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:39:46.584728 19543 catalog_manager.cc:1347] Generated new cluster ID: cb9e124d34b84506af675ad9b4f5058b
I20240506 15:39:46.584784 19543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:39:46 dist-test-slave-v0k1 krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:46.593353 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:46.581850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.17.240.254:39559)
0506 15:39:46.582072 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:46.582084 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:46.582162 (+ 78us) server_negotiation.cc:407] Connection header received
0506 15:39:46.582310 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:46.582316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:46.582365 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:46.582519 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:46.583611 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.584585 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:46.585313 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.585569 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:46.589641 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:46.589648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:46.589649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:46.589671 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:46.591780 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:46.592185 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:46.592188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:46.592188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:46.592243 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:46.592389 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:46.592390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:46.592390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:46.592627 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:46.592760 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:46.592893 (+ 133us) server_negotiation.cc:299] Negotiation successful
0506 15:39:46.593124 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":101,"threads_started":1}
I20240506 15:39:46.603776 19543 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:39:46.604364 19543 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:39:46.613425 19543 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67: Generated new TSK 0
I20240506 15:39:46.613619 19543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:39:46.638913 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39559
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:46.719324 19558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:46.719517 19558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:46.719540 19558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:46.720865 19558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:46.720916 19558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:46.720937 19558 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240506 15:39:46.720954 19558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:39:46.722559 19558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.240.254:39559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:46.722803 19558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:46.723054 19558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:46.728128 19563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:46.728806 19564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:46.729509 19566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:46.730221 19558 server_base.cc:1034] running on GCE node
I20240506 15:39:46.730414 19558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:46.730638 19558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:46.731786 19558 hybrid_clock.cc:648] HybridClock initialized: now 1715009986731768 us; error 39 us; skew 500 ppm
May 06 15:39:46 dist-test-slave-v0k1 krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009986, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:46.734349 19558 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:39:46.734753 19558 webserver.cc:460] Webserver started at http://127.17.240.193:38961/ using document root <none> and password file <none>
I20240506 15:39:46.734941 19558 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:46.734987 19558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:46.735118 19558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:46.736116 19558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "31844744f94a454aafee8ccb78424696"
format_stamp: "Formatted at 2024-05-06 15:39:46 on dist-test-slave-v0k1"
server_key: "86364817ad31baa5e8c0bddbf6749d88"
server_key_iv: "68ccf19354c8c8a237e333ac4d066fef"
server_key_version: "encryptionkey@0"
I20240506 15:39:46.736433 19558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "31844744f94a454aafee8ccb78424696"
format_stamp: "Formatted at 2024-05-06 15:39:46 on dist-test-slave-v0k1"
server_key: "86364817ad31baa5e8c0bddbf6749d88"
server_key_iv: "68ccf19354c8c8a237e333ac4d066fef"
server_key_version: "encryptionkey@0"
I20240506 15:39:46.737668 19558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:39:46.738353 19573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:46.738569 19558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:46.738680 19558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "31844744f94a454aafee8ccb78424696"
format_stamp: "Formatted at 2024-05-06 15:39:46 on dist-test-slave-v0k1"
server_key: "86364817ad31baa5e8c0bddbf6749d88"
server_key_iv: "68ccf19354c8c8a237e333ac4d066fef"
server_key_version: "encryptionkey@0"
I20240506 15:39:46.738742 19558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:46.776963 19558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:46.778918 19558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:46.779096 19558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:46.779646 19558 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:46.779687 19558 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:46.779721 19558 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:46.779740 19558 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 06 15:39:46 dist-test-slave-v0k1 krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009986, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:46.786437 19558 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:37891
I20240506 15:39:46.786779 19558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:39:46.786942 19688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:37891 every 8 connection(s)
I20240506 15:39:46.789680 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:46.780094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:46637 (local address 127.17.240.254:39559)
0506 15:39:46.780172 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:46.780176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:46.780309 (+ 133us) server_negotiation.cc:407] Connection header received
0506 15:39:46.780840 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:46.780842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:46.780861 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:46.780949 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:46.781900 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.782726 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:46.783953 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.784199 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:46.786874 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:46.786880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:46.786880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:46.786890 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:46.788519 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:46.788884 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:46.788886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:46.788886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:46.788911 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:46.789076 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:46.789076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:46.789077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:46.789179 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:46.789239 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:46.789474 (+ 235us) server_negotiation.cc:299] Negotiation successful
0506 15:39:46.789578 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240506 15:39:46.789775 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:46.780031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39559 (local address 127.17.240.193:46637)
0506 15:39:46.780191 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:46.780219 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:46.780738 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:46.781012 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:46.781015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:46.781361 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:46.781776 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:46.781783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.782831 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:46.782833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:46.783529 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:46.783532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.783694 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:46.784319 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:46.784342 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:46.786769 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:46.788605 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:46.788610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:46.788613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:46.788817 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:46.788967 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:46.788967 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:46.788968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:46.789023 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:46.789235 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:46.789240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:46.789354 (+ 114us) client_negotiation.cc:769] Sending connection context
0506 15:39:46.789447 (+ 93us) client_negotiation.cc:240] Negotiation successful
0506 15:39:46.789584 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20240506 15:39:46.790270 19689 heartbeater.cc:347] Connected to a master server at 127.17.240.254:39559
I20240506 15:39:46.790354 19689 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:46.790565 19689 heartbeater.cc:511] Master 127.17.240.254:39559 requested a full tablet report, sending...
I20240506 15:39:46.791226 19485 ts_manager.cc:194] Registered new tserver with Master: 31844744f94a454aafee8ccb78424696 (127.17.240.193:37891)
I20240506 15:39:46.791997 19485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:46637
I20240506 15:39:46.793838 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 19558
I20240506 15:39:46.793942 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:39:46.794137 18371 external_mini_cluster.cc:1428] Setting key ac1c623d871b908fc2ea97f1dc5eb7a2
I20240506 15:39:46.796634 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:46.791266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39559 (local address 127.17.240.193:60405)
0506 15:39:46.791408 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:46.791420 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:46.791497 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:46.791626 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:46.791626 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:46.791789 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:46.791976 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:46.791977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.792837 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:46.792838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:46.793205 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:46.793206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.793261 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:46.793772 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:46.793775 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:46.794050 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:46.795731 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:46.795732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:46.795733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:46.795942 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:46.796089 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:46.796089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:46.796090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:46.796124 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:46.796368 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:46.796368 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:46.796408 (+ 40us) client_negotiation.cc:769] Sending connection context
0506 15:39:46.796463 (+ 55us) client_negotiation.cc:240] Negotiation successful
0506 15:39:46.796548 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20240506 15:39:46.796744 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:46.791306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:60405 (local address 127.17.240.254:39559)
0506 15:39:46.791342 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:46.791344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:46.791496 (+ 152us) server_negotiation.cc:407] Connection header received
0506 15:39:46.791556 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:46.791557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:46.791567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:46.791619 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:46.792049 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.792744 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:46.793278 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.793409 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:46.794135 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:46.794140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:46.794140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:46.794147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:46.795620 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:46.796013 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:46.796014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:46.796014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:46.796031 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:46.796176 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:46.796176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:46.796177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:46.796296 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:46.796376 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:46.796503 (+ 127us) server_negotiation.cc:299] Negotiation successful
0506 15:39:46.796595 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:39:46.822604 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39559
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:46.901515 19698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:46.901721 19698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:46.901746 19698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:46.903028 19698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:46.903071 19698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:46.903090 19698 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240506 15:39:46.903106 19698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:39:46.904661 19698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.240.254:39559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:46.904839 19698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:46.904992 19698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:46.910118 19706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:46.910568 19703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:46.911180 19704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:46.911989 19698 server_base.cc:1034] running on GCE node
I20240506 15:39:46.912204 19698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:46.912429 19698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:46.913585 19698 hybrid_clock.cc:648] HybridClock initialized: now 1715009986913565 us; error 36 us; skew 500 ppm
May 06 15:39:46 dist-test-slave-v0k1 krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009986, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:46.916268 19698 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:39:46.916682 19698 webserver.cc:460] Webserver started at http://127.17.240.194:39071/ using document root <none> and password file <none>
I20240506 15:39:46.916865 19698 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:46.916910 19698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:46.917033 19698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:46.918061 19698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "d54c3f6d15164f4ca4ed4199ded687c4"
format_stamp: "Formatted at 2024-05-06 15:39:46 on dist-test-slave-v0k1"
server_key: "4b40377fe4a50ddaf78c8e5d109b59ee"
server_key_iv: "d37a31dbc3800afeb7e83e4ff74128e9"
server_key_version: "encryptionkey@0"
I20240506 15:39:46.918380 19698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "d54c3f6d15164f4ca4ed4199ded687c4"
format_stamp: "Formatted at 2024-05-06 15:39:46 on dist-test-slave-v0k1"
server_key: "4b40377fe4a50ddaf78c8e5d109b59ee"
server_key_iv: "d37a31dbc3800afeb7e83e4ff74128e9"
server_key_version: "encryptionkey@0"
I20240506 15:39:46.919677 19698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:39:46.920338 19713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:46.920498 19698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:46.920583 19698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "d54c3f6d15164f4ca4ed4199ded687c4"
format_stamp: "Formatted at 2024-05-06 15:39:46 on dist-test-slave-v0k1"
server_key: "4b40377fe4a50ddaf78c8e5d109b59ee"
server_key_iv: "d37a31dbc3800afeb7e83e4ff74128e9"
server_key_version: "encryptionkey@0"
I20240506 15:39:46.920642 19698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:46.932353 19698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:46.934193 19698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:46.934310 19698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:46.934814 19698 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:46.934854 19698 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:46.934885 19698 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:46.934906 19698 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:46.940680 19698 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:40305
I20240506 15:39:46.940693 19828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:40305 every 8 connection(s)
May 06 15:39:46 dist-test-slave-v0k1 krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009986, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:46.941074 19698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:39:46.944013 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:46.935260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:54529 (local address 127.17.240.254:39559)
0506 15:39:46.935334 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:46.935337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:46.935464 (+ 127us) server_negotiation.cc:407] Connection header received
0506 15:39:46.935949 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:46.935950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:46.935966 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:46.936060 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:46.937118 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.937913 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:46.938729 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.938971 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:46.941300 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:46.941305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:46.941305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:46.941313 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:46.942685 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:46.943134 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:46.943135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:46.943136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:46.943162 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:46.943381 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:46.943382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:46.943382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:46.943505 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:46.943569 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:46.943825 (+ 256us) server_negotiation.cc:299] Negotiation successful
0506 15:39:46.943922 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240506 15:39:46.944132 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:46.935202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39559 (local address 127.17.240.194:54529)
0506 15:39:46.935351 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:46.935380 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:46.935845 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:46.936121 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:46.936126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:46.936559 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:46.936993 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:46.936998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.937994 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:46.937995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:46.938631 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:46.938634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.938780 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:46.939729 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:46.939742 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:46.941196 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:46.942775 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:46.942781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:46.942783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:46.943001 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:46.943230 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:46.943231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:46.943231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:46.943298 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:46.943570 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:46.943575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:46.943691 (+ 116us) client_negotiation.cc:769] Sending connection context
0506 15:39:46.943789 (+ 98us) client_negotiation.cc:240] Negotiation successful
0506 15:39:46.943924 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20240506 15:39:46.944625 19829 heartbeater.cc:347] Connected to a master server at 127.17.240.254:39559
I20240506 15:39:46.944705 19829 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:46.944921 19829 heartbeater.cc:511] Master 127.17.240.254:39559 requested a full tablet report, sending...
I20240506 15:39:46.945360 19486 ts_manager.cc:194] Registered new tserver with Master: d54c3f6d15164f4ca4ed4199ded687c4 (127.17.240.194:40305)
I20240506 15:39:46.945868 19486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:54529
I20240506 15:39:46.947326 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 19698
I20240506 15:39:46.947407 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:39:46.947592 18371 external_mini_cluster.cc:1428] Setting key 616a1d55ce8f27f0dda6a4773ab173c4
I20240506 15:39:46.951108 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:46.945659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39559 (local address 127.17.240.194:52739)
0506 15:39:46.945775 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:46.945785 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:46.945848 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:46.946022 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:46.946023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:46.946170 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:46.946428 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:46.946430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.947090 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:46.947091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:46.947494 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:46.947496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.947576 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:46.948137 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:46.948142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:46.948417 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:46.950176 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:46.950177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:46.950177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:46.950356 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:46.950526 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:46.950527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:46.950527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:46.950549 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:46.950806 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:46.950806 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:46.950842 (+ 36us) client_negotiation.cc:769] Sending connection context
0506 15:39:46.950894 (+ 52us) client_negotiation.cc:240] Negotiation successful
0506 15:39:46.950965 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20240506 15:39:46.951189 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:46.945733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:52739 (local address 127.17.240.254:39559)
0506 15:39:46.945773 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:46.945775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:46.945847 (+ 72us) server_negotiation.cc:407] Connection header received
0506 15:39:46.945903 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:46.945903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:46.945911 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:46.945979 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:46.946532 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.946988 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:46.947575 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:46.947769 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:46.948498 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:46.948502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:46.948502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:46.948508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:46.950075 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:46.950438 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:46.950440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:46.950440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:46.950465 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:46.950599 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:46.950599 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:46.950600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:46.950748 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:46.950814 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:46.950932 (+ 118us) server_negotiation.cc:299] Negotiation successful
0506 15:39:46.951047 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:39:46.975517 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39559
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:47.056037 19838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:47.056205 19838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:47.056227 19838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:47.057508 19838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:47.057557 19838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:47.057579 19838 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240506 15:39:47.057595 19838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:39:47.059237 19838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.240.254:39559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:47.059448 19838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:47.059613 19838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:47.064561 19846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:47.065251 19844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:47.066156 19843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:47.066910 19838 server_base.cc:1034] running on GCE node
I20240506 15:39:47.067143 19838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:47.067353 19838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:47.068490 19838 hybrid_clock.cc:648] HybridClock initialized: now 1715009987068457 us; error 51 us; skew 500 ppm
May 06 15:39:47 dist-test-slave-v0k1 krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009987, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:47.070972 19838 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:39:47.071404 19838 webserver.cc:460] Webserver started at http://127.17.240.195:42333/ using document root <none> and password file <none>
I20240506 15:39:47.071599 19838 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:47.071638 19838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:47.071724 19838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:47.072680 19838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "a6202a137d98479394dec6d2650e64df"
format_stamp: "Formatted at 2024-05-06 15:39:47 on dist-test-slave-v0k1"
server_key: "4e1267f3dbb88f444d1a9d5e02379b4a"
server_key_iv: "93edca2db20f6aa967213c24c2720139"
server_key_version: "encryptionkey@0"
I20240506 15:39:47.072952 19838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "a6202a137d98479394dec6d2650e64df"
format_stamp: "Formatted at 2024-05-06 15:39:47 on dist-test-slave-v0k1"
server_key: "4e1267f3dbb88f444d1a9d5e02379b4a"
server_key_iv: "93edca2db20f6aa967213c24c2720139"
server_key_version: "encryptionkey@0"
I20240506 15:39:47.074105 19838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:39:47.074741 19853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:47.074923 19838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:47.074998 19838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "a6202a137d98479394dec6d2650e64df"
format_stamp: "Formatted at 2024-05-06 15:39:47 on dist-test-slave-v0k1"
server_key: "4e1267f3dbb88f444d1a9d5e02379b4a"
server_key_iv: "93edca2db20f6aa967213c24c2720139"
server_key_version: "encryptionkey@0"
I20240506 15:39:47.075088 19838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:47.083842 19838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:47.085558 19838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:47.085651 19838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:47.086107 19838 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:47.086151 19838 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:47.086174 19838 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:47.086185 19838 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:47.092705 19838 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:44043
I20240506 15:39:47.092772 19968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:44043 every 8 connection(s)
May 06 15:39:47 dist-test-slave-v0k1 krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009987, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:47.093075 19838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:39:47.096052 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.086488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:36425 (local address 127.17.240.254:39559)
0506 15:39:47.086560 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.086562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.086715 (+ 153us) server_negotiation.cc:407] Connection header received
0506 15:39:47.087277 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.087280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.087295 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.087387 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:47.088435 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.089313 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.090155 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.090357 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.093677 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:47.093681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:47.093682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:47.093689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:47.095042 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.095382 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.095382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.095382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.095400 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.095538 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.095539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.095539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.095659 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:47.095710 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.095916 (+ 206us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.095985 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240506 15:39:47.096222 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.086453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39559 (local address 127.17.240.195:36425)
0506 15:39:47.086600 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.086628 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.087169 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.087434 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.087438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.087866 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:47.088317 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.088324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.089415 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.089417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.090051 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.090053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.090212 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.090835 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:47.090852 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:47.093579 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:47.095118 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:47.095123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:47.095126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:47.095323 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:47.095445 (+ 122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:47.095446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:47.095446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:47.095491 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:47.095707 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:47.095710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:47.095813 (+ 103us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.095897 (+ 84us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.096004 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":87,"threads_started":1}
I20240506 15:39:47.096791 19969 heartbeater.cc:347] Connected to a master server at 127.17.240.254:39559
I20240506 15:39:47.096875 19969 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:47.097102 19969 heartbeater.cc:511] Master 127.17.240.254:39559 requested a full tablet report, sending...
I20240506 15:39:47.097582 19485 ts_manager.cc:194] Registered new tserver with Master: a6202a137d98479394dec6d2650e64df (127.17.240.195:44043)
I20240506 15:39:47.098148 19485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:36425
I20240506 15:39:47.100050 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 19838
I20240506 15:39:47.100142 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:39:47.100296 18371 external_mini_cluster.cc:1428] Setting key 64384dd9f192a56e6730b774281db160
I20240506 15:39:47.101089 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:39:47.103091 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.097903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:35281 (local address 127.17.240.254:39559)
0506 15:39:47.097940 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.097942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.098057 (+ 115us) server_negotiation.cc:407] Connection header received
0506 15:39:47.098125 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.098126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.098133 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.098179 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:47.098691 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.099225 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.099747 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.099894 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.100659 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:47.100662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:47.100662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:47.100668 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:47.102066 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.102351 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.102352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.102352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.102375 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.102507 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.102507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.102508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.102616 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:47.102681 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.102897 (+ 216us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.102968 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240506 15:39:47.103144 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.097859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39559 (local address 127.17.240.195:35281)
0506 15:39:47.097984 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.097993 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.098070 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.098185 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.098186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.098343 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:47.098589 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.098591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.099301 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.099302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.099669 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.099670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.099736 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.100252 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:47.100256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:47.100572 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:47.102147 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:47.102148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:47.102149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:47.102288 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:47.102427 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:47.102428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:47.102429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:47.102455 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:47.102713 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:47.102713 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:47.102763 (+ 50us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.102843 (+ 80us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.102957 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
May 06 15:39:47 dist-test-slave-v0k1 krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 06 15:39:47 dist-test-slave-v0k1 krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:47.124428 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.117044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33530 (local address 127.17.240.254:39559)
0506 15:39:47.117114 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.117117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.117126 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:39:47.117157 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.117158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.117167 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.117216 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:47.117907 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.118400 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.119040 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.119234 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.121398 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:47.121403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:47.121403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:47.121410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:47.123074 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.123681 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.123682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.123682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.123711 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.123880 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.123881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.123881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.123984 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:47.124114 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.124198 (+ 84us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.124302 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240506 15:39:47.126488 19485 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:39:47.128118 19485 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:39:47.133875 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.130724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37891 (local address 127.0.0.1:56748)
0506 15:39:47.131071 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.131103 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.131179 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.131518 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.131522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.131524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:47.131867 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.131869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.132790 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.132792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.133559 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.133561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.133633 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.133682 (+ 49us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.133740 (+ 58us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.133784 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":67,"threads_started":1}
I20240506 15:39:47.134322 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.130734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:44043 (local address 127.0.0.1:53312)
0506 15:39:47.131086 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.131103 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.131179 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.131626 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.131627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.131628 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:47.131876 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.131878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.133226 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.133227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.134128 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.134130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.134187 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.134191 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.134220 (+ 29us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.134255 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":93,"threads_started":1}
I20240506 15:39:47.134537 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.130902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.17.240.193:37891)
0506 15:39:47.131097 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.131125 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.131275 (+ 150us) server_negotiation.cc:407] Connection header received
0506 15:39:47.131325 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.131327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.131347 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.131417 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:47.131954 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.132703 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.133743 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.134331 (+ 588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.134378 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.134417 (+ 39us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.134462 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20240506 15:39:47.135107 19623 tablet_service.cc:1467] Processing CreateTablet for tablet ea91cb622c5c4409a5e97d14644caada (DEFAULT_TABLE table=test-table [id=8e30d7d7c7544cbe82dcd98259b6fdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:47.135259 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.130946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:40305 (local address 127.0.0.1:59704)
0506 15:39:47.131274 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.131283 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.131341 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.131546 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.131547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.131548 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:47.131839 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.131841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.134101 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.134102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.134993 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.134996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.135133 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.135138 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.135169 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.135204 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":96,"threads_started":1}
I20240506 15:39:47.135259 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.130857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.17.240.195:44043)
0506 15:39:47.131204 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.131211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.131223 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:39:47.131266 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.131267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.131286 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.131374 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:47.131965 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.133142 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.134518 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.135047 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.135098 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.135131 (+ 33us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.135182 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":82,"threads_started":1}
I20240506 15:39:47.135447 19623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea91cb622c5c4409a5e97d14644caada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:47.135865 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.131071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.17.240.194:40305)
0506 15:39:47.131239 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.131246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.131354 (+ 108us) server_negotiation.cc:407] Connection header received
0506 15:39:47.131406 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.131407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.131422 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.131510 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:47.131930 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.133143 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.135164 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.135668 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.135707 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.135739 (+ 32us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.135788 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20240506 15:39:47.135805 19903 tablet_service.cc:1467] Processing CreateTablet for tablet ea91cb622c5c4409a5e97d14644caada (DEFAULT_TABLE table=test-table [id=8e30d7d7c7544cbe82dcd98259b6fdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:47.136075 19903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea91cb622c5c4409a5e97d14644caada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:47.136700 19763 tablet_service.cc:1467] Processing CreateTablet for tablet ea91cb622c5c4409a5e97d14644caada (DEFAULT_TABLE table=test-table [id=8e30d7d7c7544cbe82dcd98259b6fdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:47.136983 19763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea91cb622c5c4409a5e97d14644caada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:47.138204 19992 tablet_bootstrap.cc:492] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: Bootstrap starting.
I20240506 15:39:47.139000 19992 tablet_bootstrap.cc:654] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:47.139379 19992 log.cc:824] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:47.139983 19995 tablet_bootstrap.cc:492] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: Bootstrap starting.
I20240506 15:39:47.140074 19993 tablet_bootstrap.cc:492] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: Bootstrap starting.
I20240506 15:39:47.140168 19992 tablet_bootstrap.cc:492] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: No bootstrap required, opened a new log
I20240506 15:39:47.140242 19992 ts_tablet_manager.cc:1367] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:39:47.140841 19993 tablet_bootstrap.cc:654] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:47.140884 19995 tablet_bootstrap.cc:654] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:47.141116 19993 log.cc:824] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:47.141185 19995 log.cc:824] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:47.141780 19993 tablet_bootstrap.cc:492] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: No bootstrap required, opened a new log
I20240506 15:39:47.141839 19993 ts_tablet_manager.cc:1367] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:39:47.142005 19992 raft_consensus.cc:348] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } }
I20240506 15:39:47.142127 19992 raft_consensus.cc:374] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:47.142151 19992 raft_consensus.cc:729] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31844744f94a454aafee8ccb78424696, State: Initialized, Role: FOLLOWER
I20240506 15:39:47.142261 19992 consensus_queue.cc:260] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } }
I20240506 15:39:47.142638 19992 ts_tablet_manager.cc:1398] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 15:39:47.142771 19689 heartbeater.cc:503] Master 127.17.240.254:39559 was elected leader, sending a full tablet report...
I20240506 15:39:47.143280 19995 tablet_bootstrap.cc:492] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: No bootstrap required, opened a new log
I20240506 15:39:47.143342 19995 ts_tablet_manager.cc:1367] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240506 15:39:47.144079 19993 raft_consensus.cc:348] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } }
I20240506 15:39:47.144191 19993 raft_consensus.cc:374] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:47.144217 19993 raft_consensus.cc:729] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6202a137d98479394dec6d2650e64df, State: Initialized, Role: FOLLOWER
I20240506 15:39:47.144325 19993 consensus_queue.cc:260] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } }
I20240506 15:39:47.144562 19993 ts_tablet_manager.cc:1398] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:39:47.144599 19969 heartbeater.cc:503] Master 127.17.240.254:39559 was elected leader, sending a full tablet report...
I20240506 15:39:47.145025 19995 raft_consensus.cc:348] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } }
I20240506 15:39:47.145153 19995 raft_consensus.cc:374] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:47.145192 19995 raft_consensus.cc:729] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d54c3f6d15164f4ca4ed4199ded687c4, State: Initialized, Role: FOLLOWER
I20240506 15:39:47.145287 19995 consensus_queue.cc:260] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } }
I20240506 15:39:47.145526 19995 ts_tablet_manager.cc:1398] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:39:47.145594 19829 heartbeater.cc:503] Master 127.17.240.254:39559 was elected leader, sending a full tablet report...
W20240506 15:39:47.191596 19830 tablet.cc:2369] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:39:47.287701 19690 tablet.cc:2369] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:39:47.288069 19999 raft_consensus.cc:482] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:39:47.288169 19999 raft_consensus.cc:504] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } }
I20240506 15:39:47.288503 19999 leader_election.cc:288] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31844744f94a454aafee8ccb78424696 (127.17.240.193:37891), d54c3f6d15164f4ca4ed4199ded687c4 (127.17.240.194:40305)
I20240506 15:39:47.290915 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.288697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37891 (local address 127.17.240.195:45863)
0506 15:39:47.288751 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.288763 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.288809 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.289013 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.289013 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.289015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:47.289253 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.289255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.290086 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.290095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.290718 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.290720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.290785 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.290788 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.290811 (+ 23us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.290837 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20240506 15:39:47.291141 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.288804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:40305 (local address 127.17.240.195:56485)
0506 15:39:47.288967 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.288976 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.289046 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.289215 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.289215 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.289217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:47.289459 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.289461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.290191 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.290192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.290814 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.290816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.290887 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.290892 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.290924 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.291046 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20240506 15:39:47.291376 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.288811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:45863 (local address 127.17.240.193:37891)
0506 15:39:47.288863 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.288865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.288877 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:39:47.288907 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.288908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.288918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.289007 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:47.289335 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.290012 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.290793 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.291281 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.291289 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.291310 (+ 21us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.291331 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240506 15:39:47.291548 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.288887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:56485 (local address 127.17.240.194:40305)
0506 15:39:47.288936 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.288938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.289049 (+ 111us) server_negotiation.cc:407] Connection header received
0506 15:39:47.289138 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.289139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.289149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.289207 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:47.289529 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.290115 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.290907 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.291404 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.291411 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.291437 (+ 26us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.291487 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240506 15:39:47.291949 19783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea91cb622c5c4409a5e97d14644caada" candidate_uuid: "a6202a137d98479394dec6d2650e64df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" is_pre_election: true
I20240506 15:39:47.291976 19643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea91cb622c5c4409a5e97d14644caada" candidate_uuid: "a6202a137d98479394dec6d2650e64df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31844744f94a454aafee8ccb78424696" is_pre_election: true
I20240506 15:39:47.292101 19643 raft_consensus.cc:2454] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6202a137d98479394dec6d2650e64df in term 0.
I20240506 15:39:47.292101 19783 raft_consensus.cc:2454] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6202a137d98479394dec6d2650e64df in term 0.
I20240506 15:39:47.292326 19857 leader_election.cc:302] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31844744f94a454aafee8ccb78424696, a6202a137d98479394dec6d2650e64df; no voters:
I20240506 15:39:47.292485 19999 raft_consensus.cc:2781] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:39:47.292563 19999 raft_consensus.cc:482] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:39:47.292588 19999 raft_consensus.cc:3037] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:47.293267 19999 raft_consensus.cc:504] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } }
I20240506 15:39:47.293382 19999 leader_election.cc:288] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [CANDIDATE]: Term 1 election: Requested vote from peers 31844744f94a454aafee8ccb78424696 (127.17.240.193:37891), d54c3f6d15164f4ca4ed4199ded687c4 (127.17.240.194:40305)
I20240506 15:39:47.293577 19643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea91cb622c5c4409a5e97d14644caada" candidate_uuid: "a6202a137d98479394dec6d2650e64df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31844744f94a454aafee8ccb78424696"
I20240506 15:39:47.293598 19783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea91cb622c5c4409a5e97d14644caada" candidate_uuid: "a6202a137d98479394dec6d2650e64df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d54c3f6d15164f4ca4ed4199ded687c4"
I20240506 15:39:47.293649 19643 raft_consensus.cc:3037] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:47.293686 19783 raft_consensus.cc:3037] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:47.294309 19643 raft_consensus.cc:2454] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6202a137d98479394dec6d2650e64df in term 1.
I20240506 15:39:47.294389 19783 raft_consensus.cc:2454] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6202a137d98479394dec6d2650e64df in term 1.
I20240506 15:39:47.294488 19857 leader_election.cc:302] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31844744f94a454aafee8ccb78424696, a6202a137d98479394dec6d2650e64df; no voters:
I20240506 15:39:47.294584 19999 raft_consensus.cc:2781] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:47.294728 19999 raft_consensus.cc:686] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 1 LEADER]: Becoming Leader. State: Replica: a6202a137d98479394dec6d2650e64df, State: Running, Role: LEADER
I20240506 15:39:47.294804 19999 consensus_queue.cc:237] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } }
I20240506 15:39:47.295643 19485 catalog_manager.cc:5484] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df reported cstate change: term changed from 0 to 1, leader changed from <none> to a6202a137d98479394dec6d2650e64df (127.17.240.195). New cstate: current_term: 1 leader_uuid: "a6202a137d98479394dec6d2650e64df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } health_report { overall_health: UNKNOWN } } }
I20240506 15:39:47.317410 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.315088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.17.240.195:44043)
0506 15:39:47.315148 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.315149 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.315176 (+ 27us) server_negotiation.cc:407] Connection header received
0506 15:39:47.315219 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.315219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.315234 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.315293 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:47.315666 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.316245 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.316839 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.317014 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.317059 (+ 45us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:47.317165 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:47.317238 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.317296 (+ 58us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.317322 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240506 15:39:47.319202 19783 raft_consensus.cc:1261] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 FOLLOWER]: Refusing update from remote peer a6202a137d98479394dec6d2650e64df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:47.319200 19643 raft_consensus.cc:1261] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696 [term 1 FOLLOWER]: Refusing update from remote peer a6202a137d98479394dec6d2650e64df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:47.319519 19999 consensus_queue.cc:1035] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [LEADER]: Connected to new peer: Peer: permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:47.319855 20002 consensus_queue.cc:1035] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [LEADER]: Connected to new peer: Peer: permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:47.322124 20005 mvcc.cc:205] Tried to move back new op lower bound from 7024680908056584192 to 7024680907960045568. Current Snapshot: MvccSnapshot[applied={T|T < 7024680908056584192}]
I20240506 15:39:47.322613 20006 mvcc.cc:205] Tried to move back new op lower bound from 7024680908056584192 to 7024680907960045568. Current Snapshot: MvccSnapshot[applied={T|T < 7024680908056584192 or (T in {7024680908056584192})}]
I20240506 15:39:47.323124 20007 mvcc.cc:205] Tried to move back new op lower bound from 7024680908056584192 to 7024680907960045568. Current Snapshot: MvccSnapshot[applied={T|T < 7024680908056584192}]
I20240506 15:39:47.326089 19485 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33530:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:47.326195 19485 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33530:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:47.326938 19485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67: Sending DeleteTablet for 3 replicas of tablet ea91cb622c5c4409a5e97d14644caada
I20240506 15:39:47.327284 19903 tablet_service.cc:1514] Processing DeleteTablet for tablet ea91cb622c5c4409a5e97d14644caada with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:47 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:53312
I20240506 15:39:47.327342 19763 tablet_service.cc:1514] Processing DeleteTablet for tablet ea91cb622c5c4409a5e97d14644caada with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:47 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:59704
I20240506 15:39:47.327518 19623 tablet_service.cc:1514] Processing DeleteTablet for tablet ea91cb622c5c4409a5e97d14644caada with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:47 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:56748
I20240506 15:39:47.327559 20015 tablet_replica.cc:330] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: stopping tablet replica
I20240506 15:39:47.327639 20015 raft_consensus.cc:2229] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:39:47.327757 20015 raft_consensus.cc:2258] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:47.327903 20016 tablet_replica.cc:330] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: stopping tablet replica
I20240506 15:39:47.327982 20016 raft_consensus.cc:2229] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:47.327971 19485 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 15:39:47.328068 20016 raft_consensus.cc:2258] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 15:39:47.328110 19485 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:39:47.328465 20015 ts_tablet_manager.cc:1875] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:47.328465 20016 ts_tablet_manager.cc:1875] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:47.329444 19763 tablet_service.cc:1467] Processing CreateTablet for tablet ef6bc6012beb477b870dbb541821bd46 (DEFAULT_TABLE table=test-table [id=cf1e46e6809e455bad5e08f0bb5150ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:47.327769 20017 tablet_replica.cc:330] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: stopping tablet replica
I20240506 15:39:47.329583 19763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6bc6012beb477b870dbb541821bd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:47.329725 20017 raft_consensus.cc:2229] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:47.330296 20016 ts_tablet_manager.cc:1888] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:39:47.330367 20016 log.cc:1196] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/ea91cb622c5c4409a5e97d14644caada
I20240506 15:39:47.330520 20017 raft_consensus.cc:2258] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:47.330639 20016 ts_tablet_manager.cc:1909] T ea91cb622c5c4409a5e97d14644caada P d54c3f6d15164f4ca4ed4199ded687c4: Deleting consensus metadata
I20240506 15:39:47.331135 20017 ts_tablet_manager.cc:1875] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:47.331180 19995 tablet_bootstrap.cc:492] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4: Bootstrap starting.
I20240506 15:39:47.331571 19995 tablet_bootstrap.cc:654] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:47.331993 19473 catalog_manager.cc:4830] TS d54c3f6d15164f4ca4ed4199ded687c4 (127.17.240.194:40305): tablet ea91cb622c5c4409a5e97d14644caada (table test-table [id=8e30d7d7c7544cbe82dcd98259b6fdb8]) successfully deleted
I20240506 15:39:47.332275 19903 tablet_service.cc:1467] Processing CreateTablet for tablet ef6bc6012beb477b870dbb541821bd46 (DEFAULT_TABLE table=test-table [id=cf1e46e6809e455bad5e08f0bb5150ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:47.332389 19903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6bc6012beb477b870dbb541821bd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:47.332593 20017 ts_tablet_manager.cc:1888] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:39:47.332705 20017 log.cc:1196] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/ea91cb622c5c4409a5e97d14644caada
I20240506 15:39:47.332703 20015 ts_tablet_manager.cc:1888] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:39:47.332754 20015 log.cc:1196] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/ea91cb622c5c4409a5e97d14644caada
I20240506 15:39:47.333029 20015 ts_tablet_manager.cc:1909] T ea91cb622c5c4409a5e97d14644caada P a6202a137d98479394dec6d2650e64df: Deleting consensus metadata
I20240506 15:39:47.333029 20017 ts_tablet_manager.cc:1909] T ea91cb622c5c4409a5e97d14644caada P 31844744f94a454aafee8ccb78424696: Deleting consensus metadata
I20240506 15:39:47.333655 19993 tablet_bootstrap.cc:492] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: Bootstrap starting.
I20240506 15:39:47.333912 19623 tablet_service.cc:1467] Processing CreateTablet for tablet ef6bc6012beb477b870dbb541821bd46 (DEFAULT_TABLE table=test-table [id=cf1e46e6809e455bad5e08f0bb5150ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:47.333974 19993 tablet_bootstrap.cc:654] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:47.334009 19623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6bc6012beb477b870dbb541821bd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:47.334497 19993 tablet_bootstrap.cc:492] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: No bootstrap required, opened a new log
I20240506 15:39:47.334551 19993 ts_tablet_manager.cc:1367] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:47.334697 19993 raft_consensus.cc:348] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.334751 19993 raft_consensus.cc:374] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:47.334774 19993 raft_consensus.cc:729] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6202a137d98479394dec6d2650e64df, State: Initialized, Role: FOLLOWER
I20240506 15:39:47.334820 19993 consensus_queue.cc:260] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.334911 19993 ts_tablet_manager.cc:1398] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:47.335090 19470 catalog_manager.cc:4830] TS a6202a137d98479394dec6d2650e64df (127.17.240.195:44043): tablet ea91cb622c5c4409a5e97d14644caada (table test-table [id=8e30d7d7c7544cbe82dcd98259b6fdb8]) successfully deleted
I20240506 15:39:47.335214 19992 tablet_bootstrap.cc:492] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696: Bootstrap starting.
I20240506 15:39:47.335287 19473 catalog_manager.cc:4830] TS 31844744f94a454aafee8ccb78424696 (127.17.240.193:37891): tablet ea91cb622c5c4409a5e97d14644caada (table test-table [id=8e30d7d7c7544cbe82dcd98259b6fdb8]) successfully deleted
I20240506 15:39:47.335516 19995 tablet_bootstrap.cc:492] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4: No bootstrap required, opened a new log
I20240506 15:39:47.335559 19992 tablet_bootstrap.cc:654] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:47.335564 19995 ts_tablet_manager.cc:1367] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240506 15:39:47.335722 19995 raft_consensus.cc:348] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.335777 19995 raft_consensus.cc:374] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:47.335801 19995 raft_consensus.cc:729] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d54c3f6d15164f4ca4ed4199ded687c4, State: Initialized, Role: FOLLOWER
I20240506 15:39:47.335851 19995 consensus_queue.cc:260] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.335942 19995 ts_tablet_manager.cc:1398] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:47.336947 19992 tablet_bootstrap.cc:492] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696: No bootstrap required, opened a new log
I20240506 15:39:47.336998 19992 ts_tablet_manager.cc:1367] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:39:47.337139 19992 raft_consensus.cc:348] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.337211 19992 raft_consensus.cc:374] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:47.337232 19992 raft_consensus.cc:729] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31844744f94a454aafee8ccb78424696, State: Initialized, Role: FOLLOWER
I20240506 15:39:47.337271 19992 consensus_queue.cc:260] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.337380 19992 ts_tablet_manager.cc:1398] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20240506 15:39:47.343725 19970 tablet.cc:2369] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:39:47.563817 20000 raft_consensus.cc:482] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:39:47.563921 20000 raft_consensus.cc:504] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.564260 20000 leader_election.cc:288] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31844744f94a454aafee8ccb78424696 (127.17.240.193:37891), a6202a137d98479394dec6d2650e64df (127.17.240.195:44043)
I20240506 15:39:47.566444 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.564408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37891 (local address 127.17.240.194:43723)
0506 15:39:47.564549 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.564559 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.564641 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.564781 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.564782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.564784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:47.564978 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.564980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.565627 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.565628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.566187 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.566189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.566252 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.566257 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.566290 (+ 33us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.566327 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20240506 15:39:47.566457 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.564409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:44043 (local address 127.17.240.194:42849)
0506 15:39:47.564582 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.564592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.564663 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.564802 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.564803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.564805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:47.564984 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.564985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.565585 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.565587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.566198 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.566200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.566268 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.566272 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.566304 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.566342 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":15,"thread_start_us":51,"threads_started":1}
I20240506 15:39:47.566828 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.564479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:43723 (local address 127.17.240.193:37891)
0506 15:39:47.564524 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.564527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.564620 (+ 93us) server_negotiation.cc:407] Connection header received
0506 15:39:47.564708 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.564710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.564719 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.564760 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:47.565049 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.565513 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.566267 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.566708 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.566717 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.566742 (+ 25us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.566773 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:39:47.566879 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.564509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:42849 (local address 127.17.240.195:44043)
0506 15:39:47.564564 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.564567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.564667 (+ 100us) server_negotiation.cc:407] Connection header received
0506 15:39:47.564726 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.564727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.564743 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.564792 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:47.565057 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.565508 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.566287 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.566766 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.566774 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.566796 (+ 22us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.566830 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240506 15:39:47.567147 19914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6bc6012beb477b870dbb541821bd46" candidate_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6202a137d98479394dec6d2650e64df" is_pre_election: true
I20240506 15:39:47.567188 19643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6bc6012beb477b870dbb541821bd46" candidate_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31844744f94a454aafee8ccb78424696" is_pre_election: true
I20240506 15:39:47.567262 19914 raft_consensus.cc:2454] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d54c3f6d15164f4ca4ed4199ded687c4 in term 0.
I20240506 15:39:47.567270 19643 raft_consensus.cc:2454] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d54c3f6d15164f4ca4ed4199ded687c4 in term 0.
I20240506 15:39:47.567425 19714 leader_election.cc:302] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6202a137d98479394dec6d2650e64df, d54c3f6d15164f4ca4ed4199ded687c4; no voters:
I20240506 15:39:47.567575 20000 raft_consensus.cc:2781] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:39:47.567631 20000 raft_consensus.cc:482] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:39:47.567646 20000 raft_consensus.cc:3037] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:47.568274 20000 raft_consensus.cc:504] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.568408 20000 leader_election.cc:288] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [CANDIDATE]: Term 1 election: Requested vote from peers 31844744f94a454aafee8ccb78424696 (127.17.240.193:37891), a6202a137d98479394dec6d2650e64df (127.17.240.195:44043)
I20240506 15:39:47.568572 19914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6bc6012beb477b870dbb541821bd46" candidate_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6202a137d98479394dec6d2650e64df"
I20240506 15:39:47.568645 19914 raft_consensus.cc:3037] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:47.568655 19643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6bc6012beb477b870dbb541821bd46" candidate_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31844744f94a454aafee8ccb78424696"
I20240506 15:39:47.568738 19643 raft_consensus.cc:3037] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:47.569289 19914 raft_consensus.cc:2454] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d54c3f6d15164f4ca4ed4199ded687c4 in term 1.
I20240506 15:39:47.569381 19643 raft_consensus.cc:2454] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d54c3f6d15164f4ca4ed4199ded687c4 in term 1.
I20240506 15:39:47.569448 19714 leader_election.cc:302] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6202a137d98479394dec6d2650e64df, d54c3f6d15164f4ca4ed4199ded687c4; no voters:
I20240506 15:39:47.569557 20000 raft_consensus.cc:2781] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:47.569692 20000 raft_consensus.cc:686] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 LEADER]: Becoming Leader. State: Replica: d54c3f6d15164f4ca4ed4199ded687c4, State: Running, Role: LEADER
I20240506 15:39:47.569756 20000 consensus_queue.cc:237] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.570408 19485 catalog_manager.cc:5484] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d54c3f6d15164f4ca4ed4199ded687c4 (127.17.240.194). New cstate: current_term: 1 leader_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } health_report { overall_health: UNKNOWN } } }
May 06 15:39:47 dist-test-slave-v0k1 krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009986, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:47.650589 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.643825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39559 (local address 127.0.0.1:33542)
0506 15:39:47.643972 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.643998 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.644087 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.644229 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.644230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.644610 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:47.644812 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.644815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.645479 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.645481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.645891 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.645893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.645964 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.646552 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:47.646556 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:47.647995 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:47.649591 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:47.649600 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:47.649604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:47.649798 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:47.649954 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:47.649955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:47.649955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:47.649982 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:47.650179 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:47.650186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:47.650231 (+ 45us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.650290 (+ 59us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.650492 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":85,"thread_start_us":57,"threads_started":1}
I20240506 15:39:47.650588 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.643937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.17.240.254:39559)
0506 15:39:47.644084 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.644086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.644099 (+ 13us) server_negotiation.cc:407] Connection header received
0506 15:39:47.644148 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.644149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.644161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.644220 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:47.644891 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.645399 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.645967 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.646139 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.648098 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:47.648105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:47.648105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:47.648111 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:47.649492 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.649866 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.649867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.649868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.649897 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.650030 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.650030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.650031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.650127 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:47.650186 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.650322 (+ 136us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.650418 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20240506 15:39:47.653110 19485 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:33542:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240506 15:39:47.653340 19485 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:39:47.654837 19623 tablet_service.cc:1467] Processing CreateTablet for tablet 61ef29156074423bbb3c6a969fe9a337 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e14e6e03b6d64e5cb68de37ec00d0ddb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240506 15:39:47.654865 19903 tablet_service.cc:1467] Processing CreateTablet for tablet 61ef29156074423bbb3c6a969fe9a337 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e14e6e03b6d64e5cb68de37ec00d0ddb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240506 15:39:47.654865 19763 tablet_service.cc:1467] Processing CreateTablet for tablet 61ef29156074423bbb3c6a969fe9a337 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e14e6e03b6d64e5cb68de37ec00d0ddb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240506 15:39:47.655000 19623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ef29156074423bbb3c6a969fe9a337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:47.655030 19903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ef29156074423bbb3c6a969fe9a337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:47.655030 19763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ef29156074423bbb3c6a969fe9a337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:47.656173 19992 tablet_bootstrap.cc:492] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: Bootstrap starting.
I20240506 15:39:47.656486 19993 tablet_bootstrap.cc:492] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: Bootstrap starting.
I20240506 15:39:47.656499 19995 tablet_bootstrap.cc:492] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: Bootstrap starting.
I20240506 15:39:47.656817 19992 tablet_bootstrap.cc:654] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:47.656881 19993 tablet_bootstrap.cc:654] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:47.656915 19995 tablet_bootstrap.cc:654] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:47.657496 19993 tablet_bootstrap.cc:492] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: No bootstrap required, opened a new log
I20240506 15:39:47.657557 19993 ts_tablet_manager.cc:1367] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:47.657656 19992 tablet_bootstrap.cc:492] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: No bootstrap required, opened a new log
I20240506 15:39:47.657709 19992 ts_tablet_manager.cc:1367] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:39:47.657689 19993 raft_consensus.cc:348] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.657773 19993 raft_consensus.cc:374] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:47.657806 19993 raft_consensus.cc:729] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6202a137d98479394dec6d2650e64df, State: Initialized, Role: FOLLOWER
I20240506 15:39:47.657831 19995 tablet_bootstrap.cc:492] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: No bootstrap required, opened a new log
I20240506 15:39:47.657855 19993 consensus_queue.cc:260] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.657853 19992 raft_consensus.cc:348] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.657888 19995 ts_tablet_manager.cc:1367] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:47.657917 19992 raft_consensus.cc:374] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:47.657944 19992 raft_consensus.cc:729] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31844744f94a454aafee8ccb78424696, State: Initialized, Role: FOLLOWER
I20240506 15:39:47.657949 19993 ts_tablet_manager.cc:1398] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:47.657999 19992 consensus_queue.cc:260] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.657994 20002 tablet_replica.cc:439] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } }
I20240506 15:39:47.658046 19995 raft_consensus.cc:348] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.658085 20002 tablet_replica.cc:442] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:47.658098 19992 ts_tablet_manager.cc:1398] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:47.658108 19995 raft_consensus.cc:374] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:47.658131 19995 raft_consensus.cc:729] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d54c3f6d15164f4ca4ed4199ded687c4, State: Initialized, Role: FOLLOWER
I20240506 15:39:47.658181 19995 consensus_queue.cc:260] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.658159 19998 tablet_replica.cc:439] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } }
I20240506 15:39:47.658224 19998 tablet_replica.cc:442] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:47.658273 19995 ts_tablet_manager.cc:1398] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:47.658295 20000 tablet_replica.cc:439] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } }
I20240506 15:39:47.658385 20000 tablet_replica.cc:442] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:47.691627 19998 raft_consensus.cc:482] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:39:47.691735 19998 raft_consensus.cc:504] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.692049 19998 leader_election.cc:288] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d54c3f6d15164f4ca4ed4199ded687c4 (127.17.240.194:40305), a6202a137d98479394dec6d2650e64df (127.17.240.195:44043)
I20240506 15:39:47.694442 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.692257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:44043 (local address 127.17.240.193:34169)
0506 15:39:47.692483 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.692495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.692609 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.692784 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.692785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.692789 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:47.692999 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.693001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.693591 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.693594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.694167 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.694169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.694229 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.694232 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.694262 (+ 30us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.694298 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
I20240506 15:39:47.694456 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.692281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:40305 (local address 127.17.240.193:52395)
0506 15:39:47.692518 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.692527 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.692576 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.692720 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.692721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.692723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:47.692952 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.692954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.693635 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.693635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.694202 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.694203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.694277 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.694280 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.694311 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.694345 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20240506 15:39:47.694850 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.692305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:34169 (local address 127.17.240.195:44043)
0506 15:39:47.692413 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.692415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.692557 (+ 142us) server_negotiation.cc:407] Connection header received
0506 15:39:47.692699 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.692699 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.692710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.692764 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:47.693085 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.693514 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.694259 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.694704 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.694712 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.694737 (+ 25us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.694771 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20240506 15:39:47.694860 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.692505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:52395 (local address 127.17.240.194:40305)
0506 15:39:47.692550 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.692552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.692562 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:39:47.692648 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.692649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.692659 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.692712 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:47.693021 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.693511 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.694292 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.694729 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.694735 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.694761 (+ 26us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.694793 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240506 15:39:47.695087 19783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ef29156074423bbb3c6a969fe9a337" candidate_uuid: "31844744f94a454aafee8ccb78424696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" is_pre_election: true
I20240506 15:39:47.695096 19914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ef29156074423bbb3c6a969fe9a337" candidate_uuid: "31844744f94a454aafee8ccb78424696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6202a137d98479394dec6d2650e64df" is_pre_election: true
I20240506 15:39:47.695166 19783 raft_consensus.cc:2454] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31844744f94a454aafee8ccb78424696 in term 0.
I20240506 15:39:47.695173 19914 raft_consensus.cc:2454] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31844744f94a454aafee8ccb78424696 in term 0.
I20240506 15:39:47.695354 19577 leader_election.cc:302] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31844744f94a454aafee8ccb78424696, d54c3f6d15164f4ca4ed4199ded687c4; no voters:
I20240506 15:39:47.695484 19998 raft_consensus.cc:2781] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:39:47.695542 19998 raft_consensus.cc:482] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:39:47.695577 19998 raft_consensus.cc:3037] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:47.696188 19998 raft_consensus.cc:504] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.696327 19998 leader_election.cc:288] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [CANDIDATE]: Term 1 election: Requested vote from peers d54c3f6d15164f4ca4ed4199ded687c4 (127.17.240.194:40305), a6202a137d98479394dec6d2650e64df (127.17.240.195:44043)
I20240506 15:39:47.696501 19783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ef29156074423bbb3c6a969fe9a337" candidate_uuid: "31844744f94a454aafee8ccb78424696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d54c3f6d15164f4ca4ed4199ded687c4"
I20240506 15:39:47.696502 19914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ef29156074423bbb3c6a969fe9a337" candidate_uuid: "31844744f94a454aafee8ccb78424696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6202a137d98479394dec6d2650e64df"
I20240506 15:39:47.696563 19783 raft_consensus.cc:3037] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:47.696565 19914 raft_consensus.cc:3037] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:47.697245 19783 raft_consensus.cc:2454] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31844744f94a454aafee8ccb78424696 in term 1.
I20240506 15:39:47.697396 19914 raft_consensus.cc:2454] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31844744f94a454aafee8ccb78424696 in term 1.
I20240506 15:39:47.697412 19577 leader_election.cc:302] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31844744f94a454aafee8ccb78424696, d54c3f6d15164f4ca4ed4199ded687c4; no voters:
I20240506 15:39:47.697500 19998 raft_consensus.cc:2781] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:47.697649 19998 raft_consensus.cc:686] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [term 1 LEADER]: Becoming Leader. State: Replica: 31844744f94a454aafee8ccb78424696, State: Running, Role: LEADER
I20240506 15:39:47.697721 19998 consensus_queue.cc:237] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } }
I20240506 15:39:47.698069 20037 tablet_replica.cc:439] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: TxnStatusTablet state changed. Reason: New leader 31844744f94a454aafee8ccb78424696. Latest consensus state: current_term: 1 leader_uuid: "31844744f94a454aafee8ccb78424696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } }
I20240506 15:39:47.698169 20037 tablet_replica.cc:442] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: This TxnStatusTablet replica's current role is: LEADER
I20240506 15:39:47.698352 20039 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240506 15:39:47.698464 20039 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240506 15:39:47.698448 19485 catalog_manager.cc:5484] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31844744f94a454aafee8ccb78424696 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "31844744f94a454aafee8ccb78424696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } health_report { overall_health: UNKNOWN } } }
I20240506 15:39:47.706400 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.704311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37891 (local address 127.0.0.1:56760)
0506 15:39:47.704393 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.704406 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.704465 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.704672 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.704673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.704674 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:39:47.704827 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.704828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.705403 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.705404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.705805 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.705806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.705876 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.705896 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:39:47.706214 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:39:47.706225 (+ 11us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.706276 (+ 51us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.706334 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20240506 15:39:47.706385 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.704376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.17.240.193:37891)
0506 15:39:47.704426 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.704428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.704439 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:39:47.704550 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.704551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.704597 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.704658 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:47.704891 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.705334 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.705879 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.706027 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.706059 (+ 32us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:47.706148 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:47.706217 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.706286 (+ 69us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.706308 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:39:47.707643 19783 raft_consensus.cc:1261] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 FOLLOWER]: Refusing update from remote peer 31844744f94a454aafee8ccb78424696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:47.707707 19914 raft_consensus.cc:1261] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df [term 1 FOLLOWER]: Refusing update from remote peer 31844744f94a454aafee8ccb78424696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:47.707913 20037 consensus_queue.cc:1035] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:47.708036 19998 consensus_queue.cc:1035] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:47.708822 20002 tablet_replica.cc:439] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: TxnStatusTablet state changed. Reason: New leader 31844744f94a454aafee8ccb78424696. Latest consensus state: current_term: 1 leader_uuid: "31844744f94a454aafee8ccb78424696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } }
I20240506 15:39:47.708917 20002 tablet_replica.cc:442] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:47.709059 20000 tablet_replica.cc:439] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: TxnStatusTablet state changed. Reason: New leader 31844744f94a454aafee8ccb78424696. Latest consensus state: current_term: 1 leader_uuid: "31844744f94a454aafee8ccb78424696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } }
I20240506 15:39:47.709156 20000 tablet_replica.cc:442] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:47.709544 19998 tablet_replica.cc:439] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31844744f94a454aafee8ccb78424696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } }
I20240506 15:39:47.709635 19998 tablet_replica.cc:442] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: This TxnStatusTablet replica's current role is: LEADER
I20240506 15:39:47.709867 20002 tablet_replica.cc:439] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31844744f94a454aafee8ccb78424696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } }
I20240506 15:39:47.709946 20002 tablet_replica.cc:442] T 61ef29156074423bbb3c6a969fe9a337 P a6202a137d98479394dec6d2650e64df: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:47.710855 19998 tablet_replica.cc:439] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31844744f94a454aafee8ccb78424696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } }
I20240506 15:39:47.710942 19998 tablet_replica.cc:442] T 61ef29156074423bbb3c6a969fe9a337 P 31844744f94a454aafee8ccb78424696: This TxnStatusTablet replica's current role is: LEADER
I20240506 15:39:47.711441 20000 tablet_replica.cc:439] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31844744f94a454aafee8ccb78424696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 } } peers { permanent_uuid: "d54c3f6d15164f4ca4ed4199ded687c4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40305 } } peers { permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 } } }
I20240506 15:39:47.711548 20000 tablet_replica.cc:442] T 61ef29156074423bbb3c6a969fe9a337 P d54c3f6d15164f4ca4ed4199ded687c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:47.713964 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.711610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.17.240.194:40305)
0506 15:39:47.711646 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.711648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.711693 (+ 45us) server_negotiation.cc:407] Connection header received
0506 15:39:47.711779 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.711780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.711799 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.711869 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:47.712209 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.712928 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.713438 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.713611 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.713656 (+ 45us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:47.713767 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:47.713853 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.713870 (+ 17us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.713893 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240506 15:39:47.718513 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.716361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37891 (local address 127.17.240.194:51267)
0506 15:39:47.716515 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.716526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.716622 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.716749 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.716750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.716751 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:39:47.717017 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.717018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.717611 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.717612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.718019 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.718020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.718103 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.718120 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:39:47.718340 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:39:47.718342 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.718389 (+ 47us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.718433 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20240506 15:39:47.718492 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.716426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:51267 (local address 127.17.240.193:37891)
0506 15:39:47.716505 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.716508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.716619 (+ 111us) server_negotiation.cc:407] Connection header received
0506 15:39:47.716682 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.716683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.716693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.716743 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:47.717097 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.717536 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.718095 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.718244 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.718261 (+ 17us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:47.718281 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:47.718329 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.718397 (+ 68us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.718416 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20240506 15:39:47.720779 19914 raft_consensus.cc:1261] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df [term 1 FOLLOWER]: Refusing update from remote peer d54c3f6d15164f4ca4ed4199ded687c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:47.720803 19643 raft_consensus.cc:1261] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696 [term 1 FOLLOWER]: Refusing update from remote peer d54c3f6d15164f4ca4ed4199ded687c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:47.721015 20023 consensus_queue.cc:1035] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6202a137d98479394dec6d2650e64df" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:47.721161 20000 consensus_queue.cc:1035] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31844744f94a454aafee8ccb78424696" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:47.730211 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.727373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:52641 (local address 127.17.240.194:40305)
0506 15:39:47.727414 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.727416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.727572 (+ 156us) server_negotiation.cc:407] Connection header received
0506 15:39:47.727636 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.727637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.727647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.727702 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:47.728239 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.728685 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.729710 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.729853 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.729872 (+ 19us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:47.729893 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:47.729937 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.730123 (+ 186us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.730153 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:39:47.730198 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.727324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:40305 (local address 127.17.240.193:52641)
0506 15:39:47.727494 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:47.727506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:47.727583 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:47.727759 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:47.727760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:47.727761 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:39:47.728138 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.728140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.728757 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.728758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:47.729578 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:47.729581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.729654 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.729674 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:39:47.729955 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:39:47.729956 (+ 1us) client_negotiation.cc:769] Sending connection context
0506 15:39:47.730032 (+ 76us) client_negotiation.cc:240] Negotiation successful
0506 15:39:47.730096 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20240506 15:39:47.737012 19485 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33530:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:47.737119 19485 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33530:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:47.737712 19485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4a504882e9134d2c86036cf59d6e5a67: Sending DeleteTablet for 3 replicas of tablet ef6bc6012beb477b870dbb541821bd46
I20240506 15:39:47.737962 19903 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6bc6012beb477b870dbb541821bd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:47 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:53312
I20240506 15:39:47.738080 20015 tablet_replica.cc:330] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: stopping tablet replica
I20240506 15:39:47.738091 19763 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6bc6012beb477b870dbb541821bd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:47 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:59704
I20240506 15:39:47.738157 20015 raft_consensus.cc:2229] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:47.738168 20016 tablet_replica.cc:330] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4: stopping tablet replica
I20240506 15:39:47.738200 20015 raft_consensus.cc:2258] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:47.738204 20016 raft_consensus.cc:2229] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:39:47.738277 20016 raft_consensus.cc:2258] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:47.738487 19623 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6bc6012beb477b870dbb541821bd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:47 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:56748
I20240506 15:39:47.738545 20015 ts_tablet_manager.cc:1875] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:47.738577 20017 tablet_replica.cc:330] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696: stopping tablet replica
I20240506 15:39:47.738613 20017 raft_consensus.cc:2229] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:47.738641 20017 raft_consensus.cc:2258] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:47.738956 20016 ts_tablet_manager.cc:1875] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:47.740361 20015 ts_tablet_manager.cc:1888] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:39:47.740434 20015 log.cc:1196] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/ef6bc6012beb477b870dbb541821bd46
I20240506 15:39:47.740540 20016 ts_tablet_manager.cc:1888] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:39:47.740608 20016 log.cc:1196] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/ef6bc6012beb477b870dbb541821bd46
I20240506 15:39:47.740695 20015 ts_tablet_manager.cc:1909] T ef6bc6012beb477b870dbb541821bd46 P a6202a137d98479394dec6d2650e64df: Deleting consensus metadata
I20240506 15:39:47.740834 20016 ts_tablet_manager.cc:1909] T ef6bc6012beb477b870dbb541821bd46 P d54c3f6d15164f4ca4ed4199ded687c4: Deleting consensus metadata
I20240506 15:39:47.741611 19470 catalog_manager.cc:4830] TS a6202a137d98479394dec6d2650e64df (127.17.240.195:44043): tablet ef6bc6012beb477b870dbb541821bd46 (table test-table [id=cf1e46e6809e455bad5e08f0bb5150ac]) successfully deleted
I20240506 15:39:47.741708 19473 catalog_manager.cc:4830] TS d54c3f6d15164f4ca4ed4199ded687c4 (127.17.240.194:40305): tablet ef6bc6012beb477b870dbb541821bd46 (table test-table [id=cf1e46e6809e455bad5e08f0bb5150ac]) successfully deleted
I20240506 15:39:47.741951 20017 ts_tablet_manager.cc:1875] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:47.743285 20017 ts_tablet_manager.cc:1888] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:39:47.743342 20017 log.cc:1196] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/ef6bc6012beb477b870dbb541821bd46
I20240506 15:39:47.743522 20017 ts_tablet_manager.cc:1909] T ef6bc6012beb477b870dbb541821bd46 P 31844744f94a454aafee8ccb78424696: Deleting consensus metadata
I20240506 15:39:47.743854 19473 catalog_manager.cc:4830] TS 31844744f94a454aafee8ccb78424696 (127.17.240.193:37891): tablet ef6bc6012beb477b870dbb541821bd46 (table test-table [id=cf1e46e6809e455bad5e08f0bb5150ac]) successfully deleted
May 06 15:39:47 dist-test-slave-v0k1 krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:39:47.755162 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.747309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.17.240.193:37891)
0506 15:39:47.747358 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.747360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.747517 (+ 157us) server_negotiation.cc:407] Connection header received
0506 15:39:47.747577 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.747578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.747588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.747641 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:47.748328 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.748813 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.749370 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.749572 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.751734 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:47.751746 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:47.751746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:47.751753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:47.754022 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.754332 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.754336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.754336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.754384 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.754533 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.754534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.754534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.754641 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:47.754787 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.754916 (+ 129us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.755041 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20240506 15:39:47.755473 19683 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56772
I20240506 15:39:47.761721 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:47.756280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.17.240.254:39559)
0506 15:39:47.756328 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:47.756330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:47.756514 (+ 184us) server_negotiation.cc:407] Connection header received
0506 15:39:47.756579 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:47.756580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:47.756590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:47.756644 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:47.757221 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.757750 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:47.758263 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:47.758793 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:47.759402 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:47.759405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:47.759406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:47.759412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:47.760760 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.761066 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.761067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.761067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.761092 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:47.761239 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:47.761240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:47.761241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:47.761326 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:47.761419 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:47.761512 (+ 93us) server_negotiation.cc:299] Negotiation successful
0506 15:39:47.761605 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20240506 15:39:47.762015 19485 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33548
I20240506 15:39:47.762537 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 19558
I20240506 15:39:47.768586 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 19698
I20240506 15:39:47.774726 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 19838
I20240506 15:39:47.780368 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 19454
2024-05-06T15:39:47Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5142 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:39:47 dist-test-slave-v0k1 krb5kdc[20060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:39:47 dist-test-slave-v0k1 krb5kdc[20060](info): set up 2 sockets
May 06 15:39:47 dist-test-slave-v0k1 krb5kdc[20060](info): commencing operation
krb5kdc: starting...
W20240506 15:39:48.746793 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:39:48 dist-test-slave-v0k1 krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:39:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:39:48.869622 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:40219
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:40219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:48.950139 20073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:48.950335 20073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:48.950359 20073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:48.951681 20073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:39:48.951721 20073 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240506 15:39:48.951735 20073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:48.951747 20073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:48.951759 20073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:39:48.951773 20073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:39:48.953332 20073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:40219
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:40219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:48.953505 20073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:48.953676 20073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:48.958388 20081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:48.959697 20079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:48.960496 20073 server_base.cc:1034] running on GCE node
W20240506 15:39:48.960508 20078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:48.960832 20073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:48.961068 20073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:48.962249 20073 hybrid_clock.cc:648] HybridClock initialized: now 1715009988962231 us; error 37 us; skew 500 ppm
May 06 15:39:48 dist-test-slave-v0k1 krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009988, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:48.964857 20073 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:39:48.965297 20073 webserver.cc:460] Webserver started at http://127.17.240.254:46305/ using document root <none> and password file <none>
I20240506 15:39:48.965497 20073 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:48.965543 20073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:48.965659 20073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:48.966675 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "06a46e4298684345afd5e06658b79e91"
format_stamp: "Formatted at 2024-05-06 15:39:48 on dist-test-slave-v0k1"
server_key: "c94961bbfd8c31fcc42061a1186c302f"
server_key_iv: "1465138d77c045676b7b88ea7fa9b284"
server_key_version: "encryptionkey@0"
I20240506 15:39:48.966996 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "06a46e4298684345afd5e06658b79e91"
format_stamp: "Formatted at 2024-05-06 15:39:48 on dist-test-slave-v0k1"
server_key: "c94961bbfd8c31fcc42061a1186c302f"
server_key_iv: "1465138d77c045676b7b88ea7fa9b284"
server_key_version: "encryptionkey@0"
I20240506 15:39:48.968255 20073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:39:48.968991 20088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:48.969216 20073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:39:48.969300 20073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "06a46e4298684345afd5e06658b79e91"
format_stamp: "Formatted at 2024-05-06 15:39:48 on dist-test-slave-v0k1"
server_key: "c94961bbfd8c31fcc42061a1186c302f"
server_key_iv: "1465138d77c045676b7b88ea7fa9b284"
server_key_version: "encryptionkey@0"
I20240506 15:39:48.969360 20073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:48.992465 20073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:48.994751 20073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:48.994876 20073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:48.999303 20073 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:40219
I20240506 15:39:48.999346 20150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:40219 every 8 connection(s)
I20240506 15:39:48.999670 20073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:39:49.000229 20151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:49.002447 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91: Bootstrap starting.
I20240506 15:39:49.003043 20151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:49.003288 20151 log.cc:824] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:49.003964 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91: No bootstrap required, opened a new log
I20240506 15:39:49.004465 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 20073
I20240506 15:39:49.004555 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:39:49.004729 18371 external_mini_cluster.cc:1428] Setting key e3634b91d7a61bd6ee0a4b8b32461a05
I20240506 15:39:49.006258 20151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a46e4298684345afd5e06658b79e91" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 40219 } }
I20240506 15:39:49.006415 20151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:49.006443 20151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a46e4298684345afd5e06658b79e91, State: Initialized, Role: FOLLOWER
I20240506 15:39:49.006577 20151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a46e4298684345afd5e06658b79e91" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 40219 } }
I20240506 15:39:49.006659 20151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:39:49.006693 20151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:39:49.006744 20151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:49.007398 20151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a46e4298684345afd5e06658b79e91" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 40219 } }
I20240506 15:39:49.007529 20151 leader_election.cc:302] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a46e4298684345afd5e06658b79e91; no voters:
I20240506 15:39:49.007695 20151 leader_election.cc:288] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:39:49.007858 20157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:49.007920 20151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:39:49.007997 20157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [term 1 LEADER]: Becoming Leader. State: Replica: 06a46e4298684345afd5e06658b79e91, State: Running, Role: LEADER
I20240506 15:39:49.008087 20157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a46e4298684345afd5e06658b79e91" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 40219 } }
I20240506 15:39:49.008394 20157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a46e4298684345afd5e06658b79e91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a46e4298684345afd5e06658b79e91" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 40219 } } }
I20240506 15:39:49.008481 20157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:49.008628 20157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a46e4298684345afd5e06658b79e91. Latest consensus state: current_term: 1 leader_uuid: "06a46e4298684345afd5e06658b79e91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a46e4298684345afd5e06658b79e91" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 40219 } } }
I20240506 15:39:49.008685 20157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91 [sys.catalog]: This master's current role is: LEADER
W20240506 15:39:49.010008 20172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:39:49.010109 20172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:39:49.010176 20165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:39:49.010365 20165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
May 06 15:39:49 dist-test-slave-v0k1 krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:49.011871 20165 catalog_manager.cc:1347] Generated new cluster ID: 9ea5dfff21204fc6a999eb70f5be4899
I20240506 15:39:49.011919 20165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:39:49.015726 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.005274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.17.240.254:40219)
0506 15:39:49.005482 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.005494 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.005519 (+ 25us) server_negotiation.cc:407] Connection header received
0506 15:39:49.006220 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.006229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.006290 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.006453 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:49.007379 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.008654 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.009694 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.009945 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.012132 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:49.012138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:49.012138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:49.012155 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:49.014270 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.014646 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.014649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.014649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.014711 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.014859 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.014860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.014860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.015138 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:49.015293 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.015400 (+ 107us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.015528 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
I20240506 15:39:49.020437 20165 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:39:49.020977 20165 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:39:49.027750 20165 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91: Generated new TSK 0
I20240506 15:39:49.027927 20165 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:39:49.058753 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:40219
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:49.165824 20177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:49.166042 20177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:49.166070 20177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:49.167691 20177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:49.167775 20177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:49.167811 20177 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240506 15:39:49.167840 20177 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240506 15:39:49.167873 20177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:39:49.170969 20177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.240.254:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:49.171269 20177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:49.171522 20177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:49.178345 20183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:49.178936 20185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:49.179870 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:49.180593 20177 server_base.cc:1034] running on GCE node
I20240506 15:39:49.180825 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:49.181078 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:49.182240 20177 hybrid_clock.cc:648] HybridClock initialized: now 1715009989182221 us; error 41 us; skew 500 ppm
May 06 15:39:49 dist-test-slave-v0k1 krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009989, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:49.185200 20177 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:39:49.185736 20177 webserver.cc:460] Webserver started at http://127.17.240.193:41303/ using document root <none> and password file <none>
I20240506 15:39:49.185952 20177 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:49.186000 20177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:49.186134 20177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:49.187256 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "4a652030bf4c4f3599ac22e3ecba4a55"
format_stamp: "Formatted at 2024-05-06 15:39:49 on dist-test-slave-v0k1"
server_key: "036674850a8d77579626551696c68fda"
server_key_iv: "b373d8049a9de60ab0e5cf10dd7afd6c"
server_key_version: "encryptionkey@0"
I20240506 15:39:49.187569 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "4a652030bf4c4f3599ac22e3ecba4a55"
format_stamp: "Formatted at 2024-05-06 15:39:49 on dist-test-slave-v0k1"
server_key: "036674850a8d77579626551696c68fda"
server_key_iv: "b373d8049a9de60ab0e5cf10dd7afd6c"
server_key_version: "encryptionkey@0"
I20240506 15:39:49.188915 20177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:39:49.189818 20192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:49.190009 20177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:49.190093 20177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "4a652030bf4c4f3599ac22e3ecba4a55"
format_stamp: "Formatted at 2024-05-06 15:39:49 on dist-test-slave-v0k1"
server_key: "036674850a8d77579626551696c68fda"
server_key_iv: "b373d8049a9de60ab0e5cf10dd7afd6c"
server_key_version: "encryptionkey@0"
I20240506 15:39:49.190167 20177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:49.200237 20177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:49.202396 20177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:49.202498 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:49.203106 20177 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:49.203155 20177 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:49.203186 20177 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:49.203202 20177 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:49.210394 20177 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:45247
I20240506 15:39:49.210464 20307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:45247 every 8 connection(s)
I20240506 15:39:49.210762 20177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
May 06 15:39:49 dist-test-slave-v0k1 krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009989, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:49.215442 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.203420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:48247 (local address 127.17.240.254:40219)
0506 15:39:49.203469 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.203471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.207354 (+ 3883us) server_negotiation.cc:407] Connection header received
0506 15:39:49.208166 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.208167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.208181 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.208255 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:49.209420 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.210281 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.211062 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.211270 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.213123 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:49.213127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:49.213127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:49.213132 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:49.214385 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.214744 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.214745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.214745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.214762 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.214896 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.214897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.214897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.214982 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:49.215089 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.215291 (+ 202us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.215374 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240506 15:39:49.215529 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.203386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:40219 (local address 127.17.240.193:48247)
0506 15:39:49.207203 (+ 3817us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.207245 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.208026 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.208316 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.208320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.208789 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:49.209271 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.209278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.210395 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.210396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.210917 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.210920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.211089 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.211680 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:49.211697 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:49.213040 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:49.214462 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.214467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.214469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.214684 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.214809 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.214810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.214810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.214857 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.215070 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:49.215073 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:49.215184 (+ 111us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.215274 (+ 90us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.215380 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3761,"thread_start_us":82,"threads_started":1}
I20240506 15:39:49.216070 20308 heartbeater.cc:347] Connected to a master server at 127.17.240.254:40219
I20240506 15:39:49.216152 20308 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:49.216352 20308 heartbeater.cc:511] Master 127.17.240.254:40219 requested a full tablet report, sending...
I20240506 15:39:49.216393 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 20177
I20240506 15:39:49.216507 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:39:49.216771 18371 external_mini_cluster.cc:1428] Setting key 294c5eaf20a75d7dbc0c7f3cbceca5f0
I20240506 15:39:49.217221 20105 ts_manager.cc:194] Registered new tserver with Master: 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247)
I20240506 15:39:49.217953 20105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:48247
I20240506 15:39:49.222800 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.217273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:50061 (local address 127.17.240.254:40219)
0506 15:39:49.217315 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.217316 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.217426 (+ 110us) server_negotiation.cc:407] Connection header received
0506 15:39:49.217484 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.217485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.217496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.217536 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:49.218102 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.218787 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.219423 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.219621 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.220355 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:49.220359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:49.220360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:49.220365 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:49.221760 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.222145 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.222146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.222146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.222164 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.222316 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.222317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.222317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.222398 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:49.222442 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.222652 (+ 210us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.222733 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:39:49.222862 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.217218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:40219 (local address 127.17.240.193:50061)
0506 15:39:49.217360 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.217369 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.217426 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.217583 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.217584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.217754 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:49.218011 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.218013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.218879 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.218880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.219349 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.219350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.219412 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.219910 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:49.219915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:49.220271 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:49.221851 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.221853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.221854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.222069 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.222222 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.222222 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.222223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.222259 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.222454 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:49.222455 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:49.222509 (+ 54us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.222600 (+ 91us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.222717 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:39:49.245762 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:40219
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:49.328563 20317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:49.328842 20317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:49.328888 20317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:49.330818 20317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:49.330886 20317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:49.330910 20317 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240506 15:39:49.330924 20317 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240506 15:39:49.330940 20317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:39:49.332763 20317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.240.254:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:49.332975 20317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:49.333137 20317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:49.338272 20323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:49.338981 20322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:49.339617 20325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:49.340306 20317 server_base.cc:1034] running on GCE node
I20240506 15:39:49.340507 20317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:49.340756 20317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:49.341897 20317 hybrid_clock.cc:648] HybridClock initialized: now 1715009989341886 us; error 30 us; skew 500 ppm
May 06 15:39:49 dist-test-slave-v0k1 krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009989, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:49.344415 20317 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:39:49.344798 20317 webserver.cc:460] Webserver started at http://127.17.240.194:37873/ using document root <none> and password file <none>
I20240506 15:39:49.344954 20317 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:49.344990 20317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:49.345075 20317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:49.346045 20317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "ab59a5185c7b42ab9736d0e0c6b5f561"
format_stamp: "Formatted at 2024-05-06 15:39:49 on dist-test-slave-v0k1"
server_key: "029b96425975d8edb9ed7580ace3e7e0"
server_key_iv: "b57f901fab999cdf1c1963a18aba327a"
server_key_version: "encryptionkey@0"
I20240506 15:39:49.346318 20317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "ab59a5185c7b42ab9736d0e0c6b5f561"
format_stamp: "Formatted at 2024-05-06 15:39:49 on dist-test-slave-v0k1"
server_key: "029b96425975d8edb9ed7580ace3e7e0"
server_key_iv: "b57f901fab999cdf1c1963a18aba327a"
server_key_version: "encryptionkey@0"
I20240506 15:39:49.347568 20317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:39:49.348292 20332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:49.348474 20317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:49.348546 20317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "ab59a5185c7b42ab9736d0e0c6b5f561"
format_stamp: "Formatted at 2024-05-06 15:39:49 on dist-test-slave-v0k1"
server_key: "029b96425975d8edb9ed7580ace3e7e0"
server_key_iv: "b57f901fab999cdf1c1963a18aba327a"
server_key_version: "encryptionkey@0"
I20240506 15:39:49.348599 20317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:49.356362 20317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:49.358358 20317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:49.358462 20317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:49.358906 20317 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:49.358945 20317 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:49.358969 20317 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:49.358983 20317 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 06 15:39:49 dist-test-slave-v0k1 krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009989, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:49.365535 20317 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:44207
I20240506 15:39:49.365595 20447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:44207 every 8 connection(s)
I20240506 15:39:49.365973 20317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:39:49.369129 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.359409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:58067 (local address 127.17.240.254:40219)
0506 15:39:49.359485 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.359488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.359642 (+ 154us) server_negotiation.cc:407] Connection header received
0506 15:39:49.360126 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.360128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.360140 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.360247 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:49.361317 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.362175 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.363127 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.363351 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.365987 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:49.365992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:49.365992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:49.366000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:49.367722 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.368187 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.368189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.368189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.368217 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.368464 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.368465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.368466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.368596 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:49.368665 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.368973 (+ 308us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.369063 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240506 15:39:49.369190 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.359358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:40219 (local address 127.17.240.194:58067)
0506 15:39:49.359528 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.359556 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.360025 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.360309 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.360313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.360733 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:49.361196 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.361203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.362282 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.362283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.363031 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.363035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.363213 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.363903 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:49.363922 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:49.365877 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:49.367819 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.367825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.367828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.368049 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.368300 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.368301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.368301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.368369 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.368688 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:49.368693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:49.368814 (+ 121us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.368926 (+ 112us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.369046 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20240506 15:39:49.370014 20448 heartbeater.cc:347] Connected to a master server at 127.17.240.254:40219
I20240506 15:39:49.370105 20448 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:49.370326 20448 heartbeater.cc:511] Master 127.17.240.254:40219 requested a full tablet report, sending...
I20240506 15:39:49.370591 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 20317
I20240506 15:39:49.370703 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:39:49.370911 18371 external_mini_cluster.cc:1428] Setting key 28b1bc68735ff2c793c75faa86c9cdca
I20240506 15:39:49.371199 20104 ts_manager.cc:194] Registered new tserver with Master: ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207)
I20240506 15:39:49.371640 20104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:58067
I20240506 15:39:49.378216 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.371216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:33183 (local address 127.17.240.254:40219)
0506 15:39:49.371271 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.371273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.371960 (+ 687us) server_negotiation.cc:407] Connection header received
0506 15:39:49.372316 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.372317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.372330 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.372393 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:49.372986 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.373453 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.374185 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.374317 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.375173 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:49.375177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:49.375178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:49.375186 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:49.376983 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.377397 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.377398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.377399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.377425 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.377656 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.377657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.377657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.377788 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:49.377854 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.378055 (+ 201us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.378143 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240506 15:39:49.378268 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.371164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:40219 (local address 127.17.240.194:33183)
0506 15:39:49.371891 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.371901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.372211 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.372507 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.372508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.372696 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:49.372891 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.372893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.373526 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.373527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.374081 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.374083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.374160 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.374723 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:49.374727 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:49.375003 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:49.377083 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.377085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.377085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.377309 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.377488 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.377489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.377490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.377526 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.377861 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:49.377862 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:49.377920 (+ 58us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.378007 (+ 87us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.378146 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:39:49.402552 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:40219
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:49.481938 20457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:49.482141 20457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:49.482167 20457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:49.483537 20457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:49.483590 20457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:49.483611 20457 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240506 15:39:49.483623 20457 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240506 15:39:49.483640 20457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:39:49.485216 20457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.240.254:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:49.485427 20457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:49.485635 20457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:49.490546 20465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:49.491279 20462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:49.492066 20463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:49.492794 20457 server_base.cc:1034] running on GCE node
I20240506 15:39:49.493007 20457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:49.493243 20457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:49.494397 20457 hybrid_clock.cc:648] HybridClock initialized: now 1715009989494392 us; error 41 us; skew 500 ppm
May 06 15:39:49 dist-test-slave-v0k1 krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009989, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:49.496958 20457 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:39:49.497380 20457 webserver.cc:460] Webserver started at http://127.17.240.195:41255/ using document root <none> and password file <none>
I20240506 15:39:49.497579 20457 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:49.497627 20457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:49.497741 20457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:49.498776 20457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "df44d35c45d14052bffed7d7411963ac"
format_stamp: "Formatted at 2024-05-06 15:39:49 on dist-test-slave-v0k1"
server_key: "8cc11bf4df6703cb6f3ad9b635289e91"
server_key_iv: "141f31921ffc04047ae803537e2e28ae"
server_key_version: "encryptionkey@0"
I20240506 15:39:49.499125 20457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "df44d35c45d14052bffed7d7411963ac"
format_stamp: "Formatted at 2024-05-06 15:39:49 on dist-test-slave-v0k1"
server_key: "8cc11bf4df6703cb6f3ad9b635289e91"
server_key_iv: "141f31921ffc04047ae803537e2e28ae"
server_key_version: "encryptionkey@0"
I20240506 15:39:49.500311 20457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:39:49.501021 20472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:49.501194 20457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:39:49.501287 20457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "df44d35c45d14052bffed7d7411963ac"
format_stamp: "Formatted at 2024-05-06 15:39:49 on dist-test-slave-v0k1"
server_key: "8cc11bf4df6703cb6f3ad9b635289e91"
server_key_iv: "141f31921ffc04047ae803537e2e28ae"
server_key_version: "encryptionkey@0"
I20240506 15:39:49.501346 20457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:49.510893 20457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:49.512746 20457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:49.512876 20457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:49.513367 20457 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:49.513407 20457 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:49.513442 20457 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:49.513475 20457 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 06 15:39:49 dist-test-slave-v0k1 krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009989, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:49.520346 20457 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:38927
I20240506 15:39:49.520393 20587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:38927 every 8 connection(s)
I20240506 15:39:49.520725 20457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:39:49.522838 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.513717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:58225 (local address 127.17.240.254:40219)
0506 15:39:49.513780 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.513783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.513979 (+ 196us) server_negotiation.cc:407] Connection header received
0506 15:39:49.514446 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.514448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.514462 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.514551 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:49.515514 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.516400 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.517227 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.517403 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.520021 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:49.520026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:49.520026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:49.520033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:49.521649 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.522028 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.522029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.522030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.522058 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.522246 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.522247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.522247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.522350 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:49.522419 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.522693 (+ 274us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.522774 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:39:49.522913 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.513704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:40219 (local address 127.17.240.195:58225)
0506 15:39:49.513864 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.513892 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.514348 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.514625 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.514629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.515039 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:49.515379 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.515384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.516535 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.516536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.517134 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.517137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.517298 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.517897 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:49.517923 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:49.519927 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:49.521725 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.521729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.521731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.521926 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.522125 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.522125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.522126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.522174 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.522439 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:49.522443 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:49.522577 (+ 134us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.522672 (+ 95us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.522780 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20240506 15:39:49.523447 20588 heartbeater.cc:347] Connected to a master server at 127.17.240.254:40219
I20240506 15:39:49.523531 20588 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:49.523783 20588 heartbeater.cc:511] Master 127.17.240.254:40219 requested a full tablet report, sending...
I20240506 15:39:49.524179 20105 ts_manager.cc:194] Registered new tserver with Master: df44d35c45d14052bffed7d7411963ac (127.17.240.195:38927)
I20240506 15:39:49.524577 20105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:58225
I20240506 15:39:49.527204 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 20457
I20240506 15:39:49.527292 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:39:49.527468 18371 external_mini_cluster.cc:1428] Setting key a6eb31def54d29e14510f39c1f02b4bb
I20240506 15:39:49.528538 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:39:49.529336 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.524586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:44507 (local address 127.17.240.254:40219)
0506 15:39:49.524706 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.524708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.524717 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:39:49.524752 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.524753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.524762 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.524800 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:49.525222 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.525676 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.526149 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.526246 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.526913 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:49.526916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:49.526917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:49.526922 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:49.528382 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.528714 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.528715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.528716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.528740 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.528876 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.528877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.528877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.528959 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:49.529014 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.529162 (+ 148us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.529239 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20240506 15:39:49.529349 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.524524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:40219 (local address 127.17.240.195:44507)
0506 15:39:49.524643 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.524653 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.524704 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.524812 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.524813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.524958 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:49.525147 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.525149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.525741 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.525741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.526095 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.526095 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.526145 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.526615 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:49.526619 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:49.526841 (+ 222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:49.528463 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.528464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.528464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.528624 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.528793 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.528794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.528795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.528823 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.529009 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:49.529010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:49.529057 (+ 47us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.529130 (+ 73us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.529239 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
May 06 15:39:49 dist-test-slave-v0k1 krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 06 15:39:49 dist-test-slave-v0k1 krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:49.549759 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.543213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.17.240.254:40219)
0506 15:39:49.543275 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.543277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.543286 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:39:49.543311 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.543311 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.543321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.543371 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:49.544048 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.544515 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.545078 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.545190 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.547179 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:49.547184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:49.547185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:49.547191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:49.548591 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.548980 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.548980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.548981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.549002 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.549255 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.549256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.549256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.549372 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:49.549501 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.549608 (+ 107us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.549686 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:39:49.551796 20105 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:39:49.553220 20105 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:39:49.558669 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.555447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:45247 (local address 127.0.0.1:57068)
0506 15:39:49.555823 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.555835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.555898 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.556542 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.556543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.556545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:49.556695 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.556696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.557576 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.557577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.558385 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.558387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.558455 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.558498 (+ 43us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.558550 (+ 52us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.558589 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":4,"thread_start_us":75,"threads_started":1}
I20240506 15:39:49.559132 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.555445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:44207 (local address 127.0.0.1:51878)
0506 15:39:49.555654 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.555681 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.555758 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.556161 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.556162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.556163 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:49.556455 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.556457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.557846 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.557847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.558703 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.558706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.558769 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.558772 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.558804 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.558984 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":66,"threads_started":1}
I20240506 15:39:49.559440 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.555684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:38927 (local address 127.0.0.1:37000)
0506 15:39:49.555823 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.555832 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.555877 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.556139 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.556142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.556144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:49.556455 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.556457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.557872 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.557873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.558605 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.558607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.558670 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.558674 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.558709 (+ 35us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.558992 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20240506 15:39:49.559562 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.555768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.17.240.195:38927)
0506 15:39:49.555953 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.555961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.555971 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:39:49.556013 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.556016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.556033 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.556120 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:49.556553 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.557768 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.558712 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.559355 (+ 643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.559410 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.559443 (+ 33us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.559492 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20240506 15:39:49.559646 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.555519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.17.240.193:45247)
0506 15:39:49.556295 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.556304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.556313 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:39:49.556351 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.556353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.556371 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.556448 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:49.556776 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.557494 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.558895 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.559449 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.559497 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.559533 (+ 36us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.559575 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":85,"threads_started":1}
I20240506 15:39:49.559751 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.555518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.17.240.194:44207)
0506 15:39:49.555996 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.556003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.556011 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:39:49.556043 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.556045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.556063 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.556159 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:49.556553 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.557761 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.559067 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.559555 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.559600 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.559628 (+ 28us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.559674 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":83,"threads_started":1}
I20240506 15:39:49.560227 20233 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef53682b4cb42cba698c2bf038a7cdc (DEFAULT_TABLE table=test-table [id=8e534e5868f84d4d8c2a484b22c49535]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:49.560257 20382 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef53682b4cb42cba698c2bf038a7cdc (DEFAULT_TABLE table=test-table [id=8e534e5868f84d4d8c2a484b22c49535]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:49.560559 20382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef53682b4cb42cba698c2bf038a7cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:49.560549 20233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef53682b4cb42cba698c2bf038a7cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:49.563439 20522 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef53682b4cb42cba698c2bf038a7cdc (DEFAULT_TABLE table=test-table [id=8e534e5868f84d4d8c2a484b22c49535]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:49.563719 20612 tablet_bootstrap.cc:492] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: Bootstrap starting.
I20240506 15:39:49.563748 20522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef53682b4cb42cba698c2bf038a7cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:49.564621 20612 tablet_bootstrap.cc:654] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:49.564921 20612 log.cc:824] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:49.565281 20611 tablet_bootstrap.cc:492] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: Bootstrap starting.
I20240506 15:39:49.565614 20612 tablet_bootstrap.cc:492] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: No bootstrap required, opened a new log
I20240506 15:39:49.565660 20612 ts_tablet_manager.cc:1367] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:39:49.566123 20611 tablet_bootstrap.cc:654] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:49.566416 20611 log.cc:824] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:49.566900 20615 tablet_bootstrap.cc:492] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: Bootstrap starting.
I20240506 15:39:49.567164 20611 tablet_bootstrap.cc:492] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: No bootstrap required, opened a new log
I20240506 15:39:49.567225 20611 ts_tablet_manager.cc:1367] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:39:49.567176 20612 raft_consensus.cc:348] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.567307 20612 raft_consensus.cc:374] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:49.567330 20612 raft_consensus.cc:729] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a652030bf4c4f3599ac22e3ecba4a55, State: Initialized, Role: FOLLOWER
I20240506 15:39:49.567428 20612 consensus_queue.cc:260] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.567684 20612 ts_tablet_manager.cc:1398] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:39:49.567770 20615 tablet_bootstrap.cc:654] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:49.567792 20308 heartbeater.cc:503] Master 127.17.240.254:40219 was elected leader, sending a full tablet report...
I20240506 15:39:49.568063 20615 log.cc:824] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:49.568845 20615 tablet_bootstrap.cc:492] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: No bootstrap required, opened a new log
I20240506 15:39:49.568912 20615 ts_tablet_manager.cc:1367] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:39:49.569182 20611 raft_consensus.cc:348] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.569303 20611 raft_consensus.cc:374] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:49.569329 20611 raft_consensus.cc:729] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab59a5185c7b42ab9736d0e0c6b5f561, State: Initialized, Role: FOLLOWER
I20240506 15:39:49.569425 20611 consensus_queue.cc:260] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.569648 20611 ts_tablet_manager.cc:1398] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 15:39:49.569723 20448 heartbeater.cc:503] Master 127.17.240.254:40219 was elected leader, sending a full tablet report...
I20240506 15:39:49.570487 20615 raft_consensus.cc:348] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.570636 20615 raft_consensus.cc:374] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:49.570657 20615 raft_consensus.cc:729] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df44d35c45d14052bffed7d7411963ac, State: Initialized, Role: FOLLOWER
I20240506 15:39:49.570748 20615 consensus_queue.cc:260] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.570995 20615 ts_tablet_manager.cc:1398] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:39:49.571033 20588 heartbeater.cc:503] Master 127.17.240.254:40219 was elected leader, sending a full tablet report...
W20240506 15:39:49.616848 20449 tablet.cc:2369] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:39:49.646677 20619 raft_consensus.cc:482] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:39:49.646813 20619 raft_consensus.cc:504] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.647207 20619 leader_election.cc:288] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207), 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247)
I20240506 15:39:49.649329 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.647416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:44207 (local address 127.17.240.195:48689)
0506 15:39:49.647478 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.647489 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.647539 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.647670 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.647671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.647672 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:49.647852 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.647854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.648501 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.648503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.649101 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.649103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.649173 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.649178 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.649210 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.649245 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20240506 15:39:49.649505 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.647424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:45247 (local address 127.17.240.195:60239)
0506 15:39:49.647655 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.647663 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.647730 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.647870 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.647870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.647872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:49.648076 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.648078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.648723 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.648724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.649287 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.649288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.649358 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.649363 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.649394 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.649427 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":11,"thread_start_us":155,"threads_started":1}
I20240506 15:39:49.649832 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.647477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:48689 (local address 127.17.240.194:44207)
0506 15:39:49.647533 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.647535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.647544 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:39:49.647604 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.647605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.647616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.647668 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:49.647923 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.648417 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.649188 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.649700 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.649708 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.649731 (+ 23us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.649772 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:39:49.649940 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.647483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:60239 (local address 127.17.240.193:45247)
0506 15:39:49.647531 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.647533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.647733 (+ 200us) server_negotiation.cc:407] Connection header received
0506 15:39:49.647798 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.647799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.647808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.647862 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:49.648147 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.648629 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.649371 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.649807 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.649813 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.649833 (+ 20us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.649853 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240506 15:39:49.650161 20402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef53682b4cb42cba698c2bf038a7cdc" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" is_pre_election: true
I20240506 15:39:49.650229 20245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef53682b4cb42cba698c2bf038a7cdc" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" is_pre_election: true
I20240506 15:39:49.650310 20402 raft_consensus.cc:2454] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 0.
I20240506 15:39:49.650381 20245 raft_consensus.cc:2454] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 0.
I20240506 15:39:49.650573 20475 leader_election.cc:302] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab59a5185c7b42ab9736d0e0c6b5f561, df44d35c45d14052bffed7d7411963ac; no voters:
I20240506 15:39:49.650718 20619 raft_consensus.cc:2781] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:39:49.650784 20619 raft_consensus.cc:482] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:39:49.650801 20619 raft_consensus.cc:3037] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:49.651808 20619 raft_consensus.cc:504] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.651953 20619 leader_election.cc:288] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 election: Requested vote from peers ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207), 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247)
I20240506 15:39:49.652125 20402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef53682b4cb42cba698c2bf038a7cdc" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561"
I20240506 15:39:49.652120 20245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef53682b4cb42cba698c2bf038a7cdc" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a652030bf4c4f3599ac22e3ecba4a55"
I20240506 15:39:49.652204 20402 raft_consensus.cc:3037] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:49.652220 20245 raft_consensus.cc:3037] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:49.653101 20245 raft_consensus.cc:2454] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 1.
I20240506 15:39:49.653101 20402 raft_consensus.cc:2454] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 1.
I20240506 15:39:49.653338 20474 leader_election.cc:302] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a652030bf4c4f3599ac22e3ecba4a55, df44d35c45d14052bffed7d7411963ac; no voters:
I20240506 15:39:49.653458 20619 raft_consensus.cc:2781] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:49.653627 20619 raft_consensus.cc:686] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 1 LEADER]: Becoming Leader. State: Replica: df44d35c45d14052bffed7d7411963ac, State: Running, Role: LEADER
I20240506 15:39:49.653724 20619 consensus_queue.cc:237] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.654541 20105 catalog_manager.cc:5484] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac reported cstate change: term changed from 0 to 1, leader changed from <none> to df44d35c45d14052bffed7d7411963ac (127.17.240.195). New cstate: current_term: 1 leader_uuid: "df44d35c45d14052bffed7d7411963ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } health_report { overall_health: HEALTHY } } }
May 06 15:39:49 dist-test-slave-v0k1 krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009988, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:49.698851 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.691910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:40219 (local address 127.0.0.1:35212)
0506 15:39:49.692096 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.692105 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.692177 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.692331 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.692332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.692713 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:49.692919 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.692922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.693618 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.693619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.694021 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.694023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.694094 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.694655 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:49.694659 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:49.696140 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:49.697903 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.697909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.697913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.698121 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.698297 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:49.698297 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:49.698298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:49.698320 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:49.698532 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:49.698538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:49.698588 (+ 50us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.698661 (+ 73us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.698772 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20240506 15:39:49.698930 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.692008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.17.240.254:40219)
0506 15:39:49.692084 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.692086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.692171 (+ 85us) server_negotiation.cc:407] Connection header received
0506 15:39:49.692237 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.692237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.692246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.692288 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:49.693041 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.693520 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.694132 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.694257 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.696238 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:49.696243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:49.696244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:49.696250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:49.697820 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.698208 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.698209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.698209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.698234 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:49.698365 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:49.698366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:49.698366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:49.698455 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:49.698505 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.698721 (+ 216us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.698827 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":49}
I20240506 15:39:49.701067 20105 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:35212:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240506 15:39:49.701290 20105 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:39:49.702740 20522 tablet_service.cc:1467] Processing CreateTablet for tablet dc95d855bfa34ddd8c66b4838ef9596e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62f60815234d4baba37f52cb2a6f828f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240506 15:39:49.702728 20233 tablet_service.cc:1467] Processing CreateTablet for tablet dc95d855bfa34ddd8c66b4838ef9596e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62f60815234d4baba37f52cb2a6f828f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240506 15:39:49.702741 20382 tablet_service.cc:1467] Processing CreateTablet for tablet dc95d855bfa34ddd8c66b4838ef9596e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62f60815234d4baba37f52cb2a6f828f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240506 15:39:49.702893 20233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc95d855bfa34ddd8c66b4838ef9596e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:49.702893 20382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc95d855bfa34ddd8c66b4838ef9596e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:49.702893 20522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc95d855bfa34ddd8c66b4838ef9596e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:49.704382 20611 tablet_bootstrap.cc:492] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: Bootstrap starting.
I20240506 15:39:49.704453 20615 tablet_bootstrap.cc:492] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: Bootstrap starting.
I20240506 15:39:49.704866 20615 tablet_bootstrap.cc:654] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:49.704866 20611 tablet_bootstrap.cc:654] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:49.705452 20612 tablet_bootstrap.cc:492] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: Bootstrap starting.
I20240506 15:39:49.705615 20615 tablet_bootstrap.cc:492] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: No bootstrap required, opened a new log
I20240506 15:39:49.705653 20615 ts_tablet_manager.cc:1367] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240506 15:39:49.705776 20615 raft_consensus.cc:348] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.705824 20615 raft_consensus.cc:374] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:49.705838 20612 tablet_bootstrap.cc:654] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:49.705860 20615 raft_consensus.cc:729] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df44d35c45d14052bffed7d7411963ac, State: Initialized, Role: FOLLOWER
I20240506 15:39:49.705896 20615 consensus_queue.cc:260] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.705994 20615 ts_tablet_manager.cc:1398] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:49.706032 20619 tablet_replica.cc:439] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } } }
I20240506 15:39:49.706125 20619 tablet_replica.cc:442] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:49.706467 20611 tablet_bootstrap.cc:492] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: No bootstrap required, opened a new log
I20240506 15:39:49.706523 20611 ts_tablet_manager.cc:1367] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:39:49.706697 20611 raft_consensus.cc:348] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.706754 20611 raft_consensus.cc:374] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:49.706777 20611 raft_consensus.cc:729] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab59a5185c7b42ab9736d0e0c6b5f561, State: Initialized, Role: FOLLOWER
I20240506 15:39:49.706827 20611 consensus_queue.cc:260] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.706924 20611 ts_tablet_manager.cc:1398] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:49.706979 20618 tablet_replica.cc:439] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } } }
I20240506 15:39:49.707101 20618 tablet_replica.cc:442] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:49.707180 20612 tablet_bootstrap.cc:492] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: No bootstrap required, opened a new log
I20240506 15:39:49.707214 20612 ts_tablet_manager.cc:1367] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:39:49.707494 20612 raft_consensus.cc:348] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.707566 20612 raft_consensus.cc:374] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:49.707600 20612 raft_consensus.cc:729] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a652030bf4c4f3599ac22e3ecba4a55, State: Initialized, Role: FOLLOWER
I20240506 15:39:49.707656 20612 consensus_queue.cc:260] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.707773 20612 ts_tablet_manager.cc:1398] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 15:39:49.707825 20616 tablet_replica.cc:439] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } } }
I20240506 15:39:49.707918 20616 tablet_replica.cc:442] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: This TxnStatusTablet replica's current role is: FOLLOWER
W20240506 15:39:49.711568 20309 tablet.cc:2369] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:39:49.771476 20589 tablet.cc:2369] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:39:49.971349 20616 raft_consensus.cc:482] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:39:49.971474 20616 raft_consensus.cc:504] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.971792 20616 leader_election.cc:288] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207), df44d35c45d14052bffed7d7411963ac (127.17.240.195:38927)
I20240506 15:39:49.974105 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.971989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:38927 (local address 127.17.240.193:41119)
0506 15:39:49.972121 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.972134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.972187 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.972363 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.972364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.972366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:49.972595 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.972597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.973264 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.973265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.973845 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.973846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.973919 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.973923 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.973954 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.973992 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20240506 15:39:49.974117 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.971989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:44207 (local address 127.17.240.193:40145)
0506 15:39:49.972134 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.972142 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.972187 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.972415 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.972416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.972419 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:39:49.972596 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.972597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.973224 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.973225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.973793 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.973795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.973864 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.973868 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.973923 (+ 55us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.973962 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":12,"thread_start_us":54,"threads_started":1}
I20240506 15:39:49.974439 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.972123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:40145 (local address 127.17.240.194:44207)
0506 15:39:49.972164 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.972166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.972180 (+ 14us) server_negotiation.cc:407] Connection header received
0506 15:39:49.972302 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.972303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.972313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.972364 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:49.972668 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.973129 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.973875 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.974316 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.974323 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.974358 (+ 35us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.974382 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240506 15:39:49.974599 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.972087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:41119 (local address 127.17.240.195:38927)
0506 15:39:49.972157 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.972159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.972202 (+ 43us) server_negotiation.cc:407] Connection header received
0506 15:39:49.972263 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.972264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.972276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.972338 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:39:49.972674 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.973162 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.973933 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.974451 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.974461 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.974485 (+ 24us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.974523 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240506 15:39:49.974669 20402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc95d855bfa34ddd8c66b4838ef9596e" candidate_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" is_pre_election: true
I20240506 15:39:49.974776 20402 raft_consensus.cc:2454] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a652030bf4c4f3599ac22e3ecba4a55 in term 0.
I20240506 15:39:49.974861 20542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc95d855bfa34ddd8c66b4838ef9596e" candidate_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df44d35c45d14052bffed7d7411963ac" is_pre_election: true
I20240506 15:39:49.974967 20542 raft_consensus.cc:2454] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a652030bf4c4f3599ac22e3ecba4a55 in term 0.
I20240506 15:39:49.974977 20195 leader_election.cc:302] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a652030bf4c4f3599ac22e3ecba4a55, ab59a5185c7b42ab9736d0e0c6b5f561; no voters:
I20240506 15:39:49.975145 20616 raft_consensus.cc:2781] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:39:49.975219 20616 raft_consensus.cc:482] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:39:49.975242 20616 raft_consensus.cc:3037] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:49.976012 20616 raft_consensus.cc:504] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.976140 20616 leader_election.cc:288] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [CANDIDATE]: Term 1 election: Requested vote from peers ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207), df44d35c45d14052bffed7d7411963ac (127.17.240.195:38927)
I20240506 15:39:49.976316 20542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc95d855bfa34ddd8c66b4838ef9596e" candidate_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df44d35c45d14052bffed7d7411963ac"
I20240506 15:39:49.976326 20402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc95d855bfa34ddd8c66b4838ef9596e" candidate_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561"
I20240506 15:39:49.976393 20402 raft_consensus.cc:3037] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:49.976397 20542 raft_consensus.cc:3037] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:49.976992 20402 raft_consensus.cc:2454] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a652030bf4c4f3599ac22e3ecba4a55 in term 1.
I20240506 15:39:49.977169 20195 leader_election.cc:302] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a652030bf4c4f3599ac22e3ecba4a55, ab59a5185c7b42ab9736d0e0c6b5f561; no voters:
I20240506 15:39:49.977167 20542 raft_consensus.cc:2454] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a652030bf4c4f3599ac22e3ecba4a55 in term 1.
I20240506 15:39:49.977335 20616 raft_consensus.cc:2781] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:49.977466 20616 raft_consensus.cc:686] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 LEADER]: Becoming Leader. State: Replica: 4a652030bf4c4f3599ac22e3ecba4a55, State: Running, Role: LEADER
I20240506 15:39:49.977559 20616 consensus_queue.cc:237] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:49.977975 20634 tablet_replica.cc:439] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: TxnStatusTablet state changed. Reason: New leader 4a652030bf4c4f3599ac22e3ecba4a55. Latest consensus state: current_term: 1 leader_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } } }
I20240506 15:39:49.978077 20634 tablet_replica.cc:442] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: This TxnStatusTablet replica's current role is: LEADER
I20240506 15:39:49.978250 20636 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240506 15:39:49.978344 20636 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240506 15:39:49.978363 20104 catalog_manager.cc:5484] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } health_report { overall_health: UNKNOWN } } }
I20240506 15:39:49.993168 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.990770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.17.240.193:45247)
0506 15:39:49.990831 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.990833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.990841 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:39:49.990874 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.990874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.990890 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.990949 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:49.991306 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.991772 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.992634 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.992773 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.992818 (+ 45us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:49.992906 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:49.992987 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:49.993082 (+ 95us) server_negotiation.cc:299] Negotiation successful
0506 15:39:49.993111 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240506 15:39:49.993178 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.990473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:45247 (local address 127.0.0.1:37726)
0506 15:39:49.990568 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:49.990584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:49.990648 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:49.990977 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:49.990978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:49.990980 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:39:49.991213 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.991221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.991843 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:49.991844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:49.992524 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:49.992527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:49.992614 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:49.992635 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:39:49.992973 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:39:49.992983 (+ 10us) client_negotiation.cc:769] Sending connection context
0506 15:39:49.993034 (+ 51us) client_negotiation.cc:240] Negotiation successful
0506 15:39:49.993091 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20240506 15:39:49.995123 20542 raft_consensus.cc:1261] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Refusing update from remote peer 4a652030bf4c4f3599ac22e3ecba4a55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:49.995126 20402 raft_consensus.cc:1261] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Refusing update from remote peer 4a652030bf4c4f3599ac22e3ecba4a55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:49.995404 20634 consensus_queue.cc:1035] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:49.995556 20616 consensus_queue.cc:1035] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:49.996591 20619 tablet_replica.cc:439] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: TxnStatusTablet state changed. Reason: New leader 4a652030bf4c4f3599ac22e3ecba4a55. Latest consensus state: current_term: 1 leader_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } } }
I20240506 15:39:49.996604 20618 tablet_replica.cc:439] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: TxnStatusTablet state changed. Reason: New leader 4a652030bf4c4f3599ac22e3ecba4a55. Latest consensus state: current_term: 1 leader_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } } }
I20240506 15:39:49.996682 20619 tablet_replica.cc:442] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:49.996683 20618 tablet_replica.cc:442] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:49.997462 20616 tablet_replica.cc:439] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } } }
I20240506 15:39:49.997552 20616 tablet_replica.cc:442] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: This TxnStatusTablet replica's current role is: LEADER
I20240506 15:39:49.998095 20619 tablet_replica.cc:439] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } } }
I20240506 15:39:49.998157 20619 tablet_replica.cc:442] T dc95d855bfa34ddd8c66b4838ef9596e P df44d35c45d14052bffed7d7411963ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:49.999317 20634 tablet_replica.cc:439] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } } }
I20240506 15:39:49.999410 20634 tablet_replica.cc:442] T dc95d855bfa34ddd8c66b4838ef9596e P 4a652030bf4c4f3599ac22e3ecba4a55: This TxnStatusTablet replica's current role is: LEADER
I20240506 15:39:49.999934 20640 mvcc.cc:205] Tried to move back new op lower bound from 7024680919017529344 to 7024680918948704256. Current Snapshot: MvccSnapshot[applied={T|T < 7024680919017529344}]
I20240506 15:39:50.000041 20618 tablet_replica.cc:439] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } } }
I20240506 15:39:50.000104 20618 tablet_replica.cc:442] T dc95d855bfa34ddd8c66b4838ef9596e P ab59a5185c7b42ab9736d0e0c6b5f561: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:39:49.999994 20641 mvcc.cc:205] Tried to move back new op lower bound from 7024680919017529344 to 7024680918948704256. Current Snapshot: MvccSnapshot[applied={T|T < 7024680919017529344}]
I20240506 15:39:50.002372 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:49.999356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.17.240.195:38927)
0506 15:39:49.999392 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:49.999394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:49.999402 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:39:49.999439 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:49.999439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:49.999456 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:49.999542 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:49.999846 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:50.001012 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:50.001875 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:50.002030 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:50.002064 (+ 34us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:50.002143 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:50.002214 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:50.002283 (+ 69us) server_negotiation.cc:299] Negotiation successful
0506 15:39:50.002305 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240506 15:39:50.006721 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:50.004853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:53567 (local address 127.17.240.193:45247)
0506 15:39:50.004910 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:50.004912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:50.004919 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:39:50.004939 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:50.004940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:50.004947 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:50.005001 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:50.005254 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:50.005734 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:50.006288 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:50.006437 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:50.006454 (+ 17us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:50.006475 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:50.006536 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:50.006623 (+ 87us) server_negotiation.cc:299] Negotiation successful
0506 15:39:50.006643 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240506 15:39:50.006721 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:50.004638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:45247 (local address 127.17.240.195:53567)
0506 15:39:50.004737 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:50.004749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:50.004792 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:50.005017 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:50.005018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:50.005020 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:39:50.005184 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:50.005186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:50.005812 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:50.005813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:50.006217 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:50.006218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:50.006282 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:50.006298 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:39:50.006567 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:39:50.006568 (+ 1us) client_negotiation.cc:769] Sending connection context
0506 15:39:50.006611 (+ 43us) client_negotiation.cc:240] Negotiation successful
0506 15:39:50.006658 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20240506 15:39:50.009347 20245 raft_consensus.cc:1261] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Refusing update from remote peer df44d35c45d14052bffed7d7411963ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:50.009335 20402 raft_consensus.cc:1261] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Refusing update from remote peer df44d35c45d14052bffed7d7411963ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:50.009663 20619 consensus_queue.cc:1035] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:50.009961 20621 consensus_queue.cc:1035] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:50.011617 20639 mvcc.cc:205] Tried to move back new op lower bound from 7024680919076327424 to 7024680917622374400. Current Snapshot: MvccSnapshot[applied={T|T < 7024680919076327424}]
I20240506 15:39:50.022965 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:50.019805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:38927 (local address 127.17.240.193:45629)
0506 15:39:50.020860 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:50.020872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:50.020954 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:50.021159 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:50.021160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:50.021161 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:39:50.021413 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:50.021416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:50.022017 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:50.022018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:50.022461 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:50.022462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:50.022529 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:50.022551 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:39:50.022782 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:39:50.022784 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:39:50.022832 (+ 48us) client_negotiation.cc:240] Negotiation successful
0506 15:39:50.022883 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"thread_start_us":102,"threads_started":1}
I20240506 15:39:50.022939 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:50.019821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:45629 (local address 127.17.240.195:38927)
0506 15:39:50.019866 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:50.019869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:50.020927 (+ 1058us) server_negotiation.cc:407] Connection header received
0506 15:39:50.021033 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:50.021034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:50.021045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:50.021097 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:50.021485 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:50.021940 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:50.022535 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:50.022683 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:50.022700 (+ 17us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:50.022721 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:50.022770 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:50.022839 (+ 69us) server_negotiation.cc:299] Negotiation successful
0506 15:39:50.022859 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:39:50.033632 20105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49806:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:50.033763 20105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49806:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:50.034533 20105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91: Sending DeleteTablet for 3 replicas of tablet 5ef53682b4cb42cba698c2bf038a7cdc
I20240506 15:39:50.034848 20233 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef53682b4cb42cba698c2bf038a7cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:50 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:57068
I20240506 15:39:50.034901 20522 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef53682b4cb42cba698c2bf038a7cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:50 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:37000
I20240506 15:39:50.035043 20382 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef53682b4cb42cba698c2bf038a7cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:50 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:51878
I20240506 15:39:50.035328 20659 tablet_replica.cc:330] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: stopping tablet replica
I20240506 15:39:50.035529 20659 raft_consensus.cc:2229] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:50.035523 20105 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 15:39:50.035645 20659 raft_consensus.cc:2258] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 15:39:50.035662 20105 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:39:50.036585 20659 ts_tablet_manager.cc:1875] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:50.036863 20233 tablet_service.cc:1467] Processing CreateTablet for tablet 844f19ebd69f4201b8b34e12881097d2 (DEFAULT_TABLE table=test-table [id=455124e6851b4527ad51baafe6f1b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:50.036994 20233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844f19ebd69f4201b8b34e12881097d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:50.037974 20612 tablet_bootstrap.cc:492] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55: Bootstrap starting.
I20240506 15:39:50.038169 20382 tablet_service.cc:1467] Processing CreateTablet for tablet 844f19ebd69f4201b8b34e12881097d2 (DEFAULT_TABLE table=test-table [id=455124e6851b4527ad51baafe6f1b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:50.038282 20658 tablet_replica.cc:330] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: stopping tablet replica
I20240506 15:39:50.038344 20522 tablet_service.cc:1467] Processing CreateTablet for tablet 844f19ebd69f4201b8b34e12881097d2 (DEFAULT_TABLE table=test-table [id=455124e6851b4527ad51baafe6f1b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:50.038374 20658 raft_consensus.cc:2229] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:39:50.038393 20612 tablet_bootstrap.cc:654] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:50.038444 20522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844f19ebd69f4201b8b34e12881097d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:50.038466 20658 raft_consensus.cc:2258] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:50.038626 20382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844f19ebd69f4201b8b34e12881097d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:50.039690 20615 tablet_bootstrap.cc:492] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac: Bootstrap starting.
I20240506 15:39:50.039798 20611 tablet_bootstrap.cc:492] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561: Bootstrap starting.
I20240506 15:39:50.040107 20615 tablet_bootstrap.cc:654] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:50.040135 20611 tablet_bootstrap.cc:654] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:50.040540 20659 ts_tablet_manager.cc:1888] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:39:50.040620 20659 log.cc:1196] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/5ef53682b4cb42cba698c2bf038a7cdc
I20240506 15:39:50.040894 20659 ts_tablet_manager.cc:1909] T 5ef53682b4cb42cba698c2bf038a7cdc P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting consensus metadata
I20240506 15:39:50.041255 20091 catalog_manager.cc:4830] TS ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207): tablet 5ef53682b4cb42cba698c2bf038a7cdc (table test-table [id=8e534e5868f84d4d8c2a484b22c49535]) successfully deleted
I20240506 15:39:50.041316 20615 tablet_bootstrap.cc:492] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac: No bootstrap required, opened a new log
I20240506 15:39:50.041359 20615 ts_tablet_manager.cc:1367] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:39:50.041414 20657 tablet_replica.cc:330] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: stopping tablet replica
I20240506 15:39:50.041498 20657 raft_consensus.cc:2229] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:50.041546 20657 raft_consensus.cc:2258] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:50.041528 20615 raft_consensus.cc:348] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:50.041592 20615 raft_consensus.cc:374] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:50.041612 20615 raft_consensus.cc:729] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df44d35c45d14052bffed7d7411963ac, State: Initialized, Role: FOLLOWER
I20240506 15:39:50.041656 20615 consensus_queue.cc:260] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:50.041754 20615 ts_tablet_manager.cc:1398] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:50.042366 20657 ts_tablet_manager.cc:1875] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:50.043520 20612 tablet_bootstrap.cc:492] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55: No bootstrap required, opened a new log
I20240506 15:39:50.043574 20612 ts_tablet_manager.cc:1367] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240506 15:39:50.043615 20658 ts_tablet_manager.cc:1875] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:50.043720 20612 raft_consensus.cc:348] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:50.043782 20612 raft_consensus.cc:374] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:50.043802 20612 raft_consensus.cc:729] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a652030bf4c4f3599ac22e3ecba4a55, State: Initialized, Role: FOLLOWER
I20240506 15:39:50.043845 20612 consensus_queue.cc:260] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:50.043934 20612 ts_tablet_manager.cc:1398] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:50.046397 20657 ts_tablet_manager.cc:1888] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:39:50.046468 20657 log.cc:1196] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/5ef53682b4cb42cba698c2bf038a7cdc
I20240506 15:39:50.046743 20657 ts_tablet_manager.cc:1909] T 5ef53682b4cb42cba698c2bf038a7cdc P 4a652030bf4c4f3599ac22e3ecba4a55: Deleting consensus metadata
I20240506 15:39:50.046926 20658 ts_tablet_manager.cc:1888] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:39:50.046978 20658 log.cc:1196] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/5ef53682b4cb42cba698c2bf038a7cdc
I20240506 15:39:50.047247 20658 ts_tablet_manager.cc:1909] T 5ef53682b4cb42cba698c2bf038a7cdc P df44d35c45d14052bffed7d7411963ac: Deleting consensus metadata
I20240506 15:39:50.047628 20092 catalog_manager.cc:4830] TS df44d35c45d14052bffed7d7411963ac (127.17.240.195:38927): tablet 5ef53682b4cb42cba698c2bf038a7cdc (table test-table [id=8e534e5868f84d4d8c2a484b22c49535]) successfully deleted
I20240506 15:39:50.047971 20090 catalog_manager.cc:4830] TS 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247): tablet 5ef53682b4cb42cba698c2bf038a7cdc (table test-table [id=8e534e5868f84d4d8c2a484b22c49535]) successfully deleted
I20240506 15:39:50.048432 20611 tablet_bootstrap.cc:492] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561: No bootstrap required, opened a new log
I20240506 15:39:50.048480 20611 ts_tablet_manager.cc:1367] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240506 15:39:50.048640 20611 raft_consensus.cc:348] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:50.048705 20611 raft_consensus.cc:374] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:50.048724 20611 raft_consensus.cc:729] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab59a5185c7b42ab9736d0e0c6b5f561, State: Initialized, Role: FOLLOWER
I20240506 15:39:50.048768 20611 consensus_queue.cc:260] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:50.048849 20611 ts_tablet_manager.cc:1398] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:50.127369 20621 raft_consensus.cc:482] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:39:50.127502 20621 raft_consensus.cc:504] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:50.127686 20621 leader_election.cc:288] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247), ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207)
I20240506 15:39:50.127947 20402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844f19ebd69f4201b8b34e12881097d2" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" is_pre_election: true
I20240506 15:39:50.127974 20245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844f19ebd69f4201b8b34e12881097d2" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" is_pre_election: true
I20240506 15:39:50.128054 20402 raft_consensus.cc:2454] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 0.
I20240506 15:39:50.128067 20245 raft_consensus.cc:2454] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 0.
I20240506 15:39:50.128242 20474 leader_election.cc:302] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a652030bf4c4f3599ac22e3ecba4a55, df44d35c45d14052bffed7d7411963ac; no voters:
I20240506 15:39:50.128332 20621 raft_consensus.cc:2781] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:39:50.128367 20621 raft_consensus.cc:482] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:39:50.128386 20621 raft_consensus.cc:3037] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:50.129293 20621 raft_consensus.cc:504] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:50.129463 20621 leader_election.cc:288] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 election: Requested vote from peers 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247), ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207)
I20240506 15:39:50.129699 20402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844f19ebd69f4201b8b34e12881097d2" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561"
I20240506 15:39:50.129724 20245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844f19ebd69f4201b8b34e12881097d2" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a652030bf4c4f3599ac22e3ecba4a55"
I20240506 15:39:50.129786 20402 raft_consensus.cc:3037] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:50.129810 20245 raft_consensus.cc:3037] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:50.130656 20245 raft_consensus.cc:2454] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 1.
I20240506 15:39:50.130657 20402 raft_consensus.cc:2454] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 1.
I20240506 15:39:50.131039 20475 leader_election.cc:302] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab59a5185c7b42ab9736d0e0c6b5f561, df44d35c45d14052bffed7d7411963ac; no voters:
I20240506 15:39:50.131158 20621 raft_consensus.cc:2781] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:50.131197 20621 raft_consensus.cc:686] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 1 LEADER]: Becoming Leader. State: Replica: df44d35c45d14052bffed7d7411963ac, State: Running, Role: LEADER
I20240506 15:39:50.131253 20621 consensus_queue.cc:237] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:50.131907 20105 catalog_manager.cc:5484] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac reported cstate change: term changed from 0 to 1, leader changed from <none> to df44d35c45d14052bffed7d7411963ac (127.17.240.195). New cstate: current_term: 1 leader_uuid: "df44d35c45d14052bffed7d7411963ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } health_report { overall_health: HEALTHY } } }
I20240506 15:39:50.154860 20402 raft_consensus.cc:1261] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Refusing update from remote peer df44d35c45d14052bffed7d7411963ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:50.155126 20619 consensus_queue.cc:1035] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:50.155418 20245 raft_consensus.cc:1261] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Refusing update from remote peer df44d35c45d14052bffed7d7411963ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:50.155834 20619 consensus_queue.cc:1035] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 15:39:50.164645 20641 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240506 15:39:51.689832 20105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49806:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:51.689939 20105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49806:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:51.691051 20105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91: Sending DeleteTablet for 3 replicas of tablet 844f19ebd69f4201b8b34e12881097d2
I20240506 15:39:51.691339 20233 tablet_service.cc:1514] Processing DeleteTablet for tablet 844f19ebd69f4201b8b34e12881097d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:51 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:57068
I20240506 15:39:51.691375 20382 tablet_service.cc:1514] Processing DeleteTablet for tablet 844f19ebd69f4201b8b34e12881097d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:51 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:51878
I20240506 15:39:51.691495 20522 tablet_service.cc:1514] Processing DeleteTablet for tablet 844f19ebd69f4201b8b34e12881097d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:51 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:37000
I20240506 15:39:51.691599 20688 tablet_replica.cc:330] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561: stopping tablet replica
I20240506 15:39:51.691673 20688 raft_consensus.cc:2229] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:51.691718 20688 raft_consensus.cc:2258] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:51.691716 20689 tablet_replica.cc:330] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac: stopping tablet replica
I20240506 15:39:51.691768 20689 raft_consensus.cc:2229] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:39:51.691843 20689 raft_consensus.cc:2258] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:51.692096 20689 ts_tablet_manager.cc:1875] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:51.697005 20688 ts_tablet_manager.cc:1875] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:51.697058 20687 tablet_replica.cc:330] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55: stopping tablet replica
I20240506 15:39:51.697374 20687 raft_consensus.cc:2229] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:51.697497 20687 raft_consensus.cc:2258] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:51.697970 20687 ts_tablet_manager.cc:1875] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55: Deleting tablet data with delete state TABLET_DATA_DELETED
May 06 15:39:51 dist-test-slave-v0k1 krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20240506 15:39:51.713675 20689 ts_tablet_manager.cc:1888] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240506 15:39:51.713759 20689 log.cc:1196] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/844f19ebd69f4201b8b34e12881097d2
I20240506 15:39:51.714032 20689 ts_tablet_manager.cc:1909] T 844f19ebd69f4201b8b34e12881097d2 P df44d35c45d14052bffed7d7411963ac: Deleting consensus metadata
I20240506 15:39:51.715263 20688 ts_tablet_manager.cc:1888] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240506 15:39:51.715271 20687 ts_tablet_manager.cc:1888] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240506 15:39:51.715334 20688 log.cc:1196] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/844f19ebd69f4201b8b34e12881097d2
I20240506 15:39:51.715339 20687 log.cc:1196] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/844f19ebd69f4201b8b34e12881097d2
I20240506 15:39:51.715580 20687 ts_tablet_manager.cc:1909] T 844f19ebd69f4201b8b34e12881097d2 P 4a652030bf4c4f3599ac22e3ecba4a55: Deleting consensus metadata
I20240506 15:39:51.715605 20688 ts_tablet_manager.cc:1909] T 844f19ebd69f4201b8b34e12881097d2 P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting consensus metadata
I20240506 15:39:51.716984 20092 catalog_manager.cc:4830] TS df44d35c45d14052bffed7d7411963ac (127.17.240.195:38927): tablet 844f19ebd69f4201b8b34e12881097d2 (table test-table [id=455124e6851b4527ad51baafe6f1b2a2]) successfully deleted
I20240506 15:39:51.727043 20090 catalog_manager.cc:4830] TS 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247): tablet 844f19ebd69f4201b8b34e12881097d2 (table test-table [id=455124e6851b4527ad51baafe6f1b2a2]) successfully deleted
I20240506 15:39:51.727043 20091 catalog_manager.cc:4830] TS ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207): tablet 844f19ebd69f4201b8b34e12881097d2 (table test-table [id=455124e6851b4527ad51baafe6f1b2a2]) successfully deleted
May 06 15:39:51 dist-test-slave-v0k1 krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:51.752228 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:51.739222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.17.240.254:40219)
0506 15:39:51.739375 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:51.739377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:51.739516 (+ 139us) server_negotiation.cc:407] Connection header received
0506 15:39:51.739596 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:51.739596 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:51.739609 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:51.739672 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:51.740385 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:51.740915 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:51.741494 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:51.741693 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:51.748970 (+ 7277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:51.748978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:51.748978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:51.748985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:51.750600 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:51.751471 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:51.751472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:51.751472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:51.751499 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:51.751725 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:51.751726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:51.751726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:51.751834 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:51.751913 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:51.752016 (+ 103us) server_negotiation.cc:299] Negotiation successful
0506 15:39:51.752103 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20240506 15:39:51.753728 20105 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:39:51.753873 20105 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:39:51.755309 20233 tablet_service.cc:1467] Processing CreateTablet for tablet e1d48d4ad6d8437ba7764a47e10347f8 (DEFAULT_TABLE table=test-table [id=febc1763b554427993a6425295d2afce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:51.755344 20522 tablet_service.cc:1467] Processing CreateTablet for tablet e1d48d4ad6d8437ba7764a47e10347f8 (DEFAULT_TABLE table=test-table [id=febc1763b554427993a6425295d2afce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:51.755462 20233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d48d4ad6d8437ba7764a47e10347f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:51.755477 20522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d48d4ad6d8437ba7764a47e10347f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:51.756541 20699 tablet_bootstrap.cc:492] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55: Bootstrap starting.
I20240506 15:39:51.756574 20382 tablet_service.cc:1467] Processing CreateTablet for tablet e1d48d4ad6d8437ba7764a47e10347f8 (DEFAULT_TABLE table=test-table [id=febc1763b554427993a6425295d2afce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:51.756646 20698 tablet_bootstrap.cc:492] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac: Bootstrap starting.
I20240506 15:39:51.756691 20382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d48d4ad6d8437ba7764a47e10347f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:51.756924 20699 tablet_bootstrap.cc:654] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:51.757097 20698 tablet_bootstrap.cc:654] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:51.757771 20698 tablet_bootstrap.cc:492] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac: No bootstrap required, opened a new log
I20240506 15:39:51.757824 20698 ts_tablet_manager.cc:1367] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:51.757966 20698 raft_consensus.cc:348] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.758052 20698 raft_consensus.cc:374] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:51.758075 20698 raft_consensus.cc:729] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df44d35c45d14052bffed7d7411963ac, State: Initialized, Role: FOLLOWER
I20240506 15:39:51.758141 20702 tablet_bootstrap.cc:492] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561: Bootstrap starting.
I20240506 15:39:51.758149 20698 consensus_queue.cc:260] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.758502 20702 tablet_bootstrap.cc:654] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:51.758386 20698 ts_tablet_manager.cc:1398] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:51.759040 20699 tablet_bootstrap.cc:492] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55: No bootstrap required, opened a new log
I20240506 15:39:51.759105 20699 ts_tablet_manager.cc:1367] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240506 15:39:51.759330 20699 raft_consensus.cc:348] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.759403 20699 raft_consensus.cc:374] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:51.759418 20699 raft_consensus.cc:729] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a652030bf4c4f3599ac22e3ecba4a55, State: Initialized, Role: FOLLOWER
I20240506 15:39:51.759488 20699 consensus_queue.cc:260] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.759599 20699 ts_tablet_manager.cc:1398] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:51.759828 20702 tablet_bootstrap.cc:492] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561: No bootstrap required, opened a new log
I20240506 15:39:51.759886 20702 ts_tablet_manager.cc:1367] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:39:51.760001 20702 raft_consensus.cc:348] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.760074 20702 raft_consensus.cc:374] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:51.760098 20702 raft_consensus.cc:729] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab59a5185c7b42ab9736d0e0c6b5f561, State: Initialized, Role: FOLLOWER
I20240506 15:39:51.760145 20702 consensus_queue.cc:260] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.760331 20702 ts_tablet_manager.cc:1398] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:51.805444 20619 raft_consensus.cc:482] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:39:51.805545 20619 raft_consensus.cc:504] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.805675 20619 leader_election.cc:288] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247), ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207)
I20240506 15:39:51.805869 20245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d48d4ad6d8437ba7764a47e10347f8" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" is_pre_election: true
I20240506 15:39:51.805872 20401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d48d4ad6d8437ba7764a47e10347f8" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" is_pre_election: true
I20240506 15:39:51.805953 20245 raft_consensus.cc:2454] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 0.
I20240506 15:39:51.805976 20401 raft_consensus.cc:2454] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 0.
I20240506 15:39:51.806138 20475 leader_election.cc:302] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab59a5185c7b42ab9736d0e0c6b5f561, df44d35c45d14052bffed7d7411963ac; no voters:
I20240506 15:39:51.806241 20619 raft_consensus.cc:2781] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:39:51.806278 20619 raft_consensus.cc:482] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:39:51.806298 20619 raft_consensus.cc:3037] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:51.811220 20619 raft_consensus.cc:504] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.811422 20619 leader_election.cc:288] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 election: Requested vote from peers 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247), ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207)
I20240506 15:39:51.811671 20245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d48d4ad6d8437ba7764a47e10347f8" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a652030bf4c4f3599ac22e3ecba4a55"
I20240506 15:39:51.811669 20401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d48d4ad6d8437ba7764a47e10347f8" candidate_uuid: "df44d35c45d14052bffed7d7411963ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561"
I20240506 15:39:51.811765 20401 raft_consensus.cc:3037] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:51.811769 20245 raft_consensus.cc:3037] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:51.812369 20245 raft_consensus.cc:2454] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 1.
I20240506 15:39:51.812548 20474 leader_election.cc:302] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a652030bf4c4f3599ac22e3ecba4a55, df44d35c45d14052bffed7d7411963ac; no voters:
I20240506 15:39:51.812659 20619 raft_consensus.cc:2781] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:51.812711 20619 raft_consensus.cc:686] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 1 LEADER]: Becoming Leader. State: Replica: df44d35c45d14052bffed7d7411963ac, State: Running, Role: LEADER
I20240506 15:39:51.812758 20401 raft_consensus.cc:2454] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df44d35c45d14052bffed7d7411963ac in term 1.
I20240506 15:39:51.812767 20619 consensus_queue.cc:237] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.813403 20104 catalog_manager.cc:5484] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac reported cstate change: term changed from 0 to 1, leader changed from <none> to df44d35c45d14052bffed7d7411963ac (127.17.240.195). New cstate: current_term: 1 leader_uuid: "df44d35c45d14052bffed7d7411963ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } health_report { overall_health: HEALTHY } } }
I20240506 15:39:51.824088 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:51.822015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.17.240.195:38927)
0506 15:39:51.822174 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:51.822175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:51.822185 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:39:51.822205 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:51.822206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:51.822215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:51.822266 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:51.822504 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:51.823103 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:51.823682 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:51.823788 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:51.823803 (+ 15us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:51.823827 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:51.823881 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:51.823992 (+ 111us) server_negotiation.cc:299] Negotiation successful
0506 15:39:51.824013 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20240506 15:39:51.826646 20401 raft_consensus.cc:1261] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Refusing update from remote peer df44d35c45d14052bffed7d7411963ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:51.826725 20245 raft_consensus.cc:1261] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Refusing update from remote peer df44d35c45d14052bffed7d7411963ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:51.826948 20619 consensus_queue.cc:1035] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:51.827082 20682 consensus_queue.cc:1035] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:51.837193 20105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35216:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:51.837297 20105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35216:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:51.838034 20105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91: Sending DeleteTablet for 3 replicas of tablet e1d48d4ad6d8437ba7764a47e10347f8
I20240506 15:39:51.838307 20233 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d48d4ad6d8437ba7764a47e10347f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:51 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:57068
I20240506 15:39:51.838310 20382 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d48d4ad6d8437ba7764a47e10347f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:51 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:51878
I20240506 15:39:51.838407 20687 tablet_replica.cc:330] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55: stopping tablet replica
I20240506 15:39:51.838464 20687 raft_consensus.cc:2229] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:51.838538 20687 raft_consensus.cc:2258] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:51.838595 20688 tablet_replica.cc:330] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561: stopping tablet replica
I20240506 15:39:51.838657 20688 raft_consensus.cc:2229] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:51.838697 20688 raft_consensus.cc:2258] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:51.838843 20687 ts_tablet_manager.cc:1875] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:51.838927 20522 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d48d4ad6d8437ba7764a47e10347f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:51 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:37000
I20240506 15:39:51.839027 20689 tablet_replica.cc:330] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac: stopping tablet replica
I20240506 15:39:51.839058 20688 ts_tablet_manager.cc:1875] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:51.839073 20689 raft_consensus.cc:2229] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:39:51.839143 20689 raft_consensus.cc:2258] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:51.839457 20689 ts_tablet_manager.cc:1875] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:51.840078 20105 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:39:51.840227 20105 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:39:51.841660 20522 tablet_service.cc:1467] Processing CreateTablet for tablet 14e1328f26e14b2eb814ae46d2b5ce8c (DEFAULT_TABLE table=test-table [id=114813407295490dab556ae08828f679]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:51.841697 20382 tablet_service.cc:1467] Processing CreateTablet for tablet 14e1328f26e14b2eb814ae46d2b5ce8c (DEFAULT_TABLE table=test-table [id=114813407295490dab556ae08828f679]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:51.841701 20233 tablet_service.cc:1467] Processing CreateTablet for tablet 14e1328f26e14b2eb814ae46d2b5ce8c (DEFAULT_TABLE table=test-table [id=114813407295490dab556ae08828f679]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:39:51.898808 20382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e1328f26e14b2eb814ae46d2b5ce8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:51.898810 20522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e1328f26e14b2eb814ae46d2b5ce8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:51.898808 20233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e1328f26e14b2eb814ae46d2b5ce8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:51.900274 20698 tablet_bootstrap.cc:492] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac: Bootstrap starting.
I20240506 15:39:51.900633 20699 tablet_bootstrap.cc:492] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55: Bootstrap starting.
I20240506 15:39:51.900713 20698 tablet_bootstrap.cc:654] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:51.900983 20699 tablet_bootstrap.cc:654] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:51.902062 20699 tablet_bootstrap.cc:492] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55: No bootstrap required, opened a new log
I20240506 15:39:51.902057 20702 tablet_bootstrap.cc:492] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561: Bootstrap starting.
I20240506 15:39:51.902127 20699 ts_tablet_manager.cc:1367] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:39:51.902261 20699 raft_consensus.cc:348] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.902348 20699 raft_consensus.cc:374] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:51.902376 20699 raft_consensus.cc:729] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a652030bf4c4f3599ac22e3ecba4a55, State: Initialized, Role: FOLLOWER
I20240506 15:39:51.902416 20702 tablet_bootstrap.cc:654] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:51.902419 20699 consensus_queue.cc:260] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.902529 20699 ts_tablet_manager.cc:1398] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:51.902889 20698 tablet_bootstrap.cc:492] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac: No bootstrap required, opened a new log
I20240506 15:39:51.902940 20698 ts_tablet_manager.cc:1367] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240506 15:39:51.903088 20698 raft_consensus.cc:348] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.903174 20698 raft_consensus.cc:374] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:51.903199 20698 raft_consensus.cc:729] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df44d35c45d14052bffed7d7411963ac, State: Initialized, Role: FOLLOWER
I20240506 15:39:51.903211 20702 tablet_bootstrap.cc:492] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561: No bootstrap required, opened a new log
I20240506 15:39:51.903260 20702 ts_tablet_manager.cc:1367] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:51.903249 20698 consensus_queue.cc:260] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.903342 20698 ts_tablet_manager.cc:1398] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:51.903489 20702 raft_consensus.cc:348] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.903576 20702 raft_consensus.cc:374] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:51.903599 20702 raft_consensus.cc:729] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab59a5185c7b42ab9736d0e0c6b5f561, State: Initialized, Role: FOLLOWER
I20240506 15:39:51.903653 20702 consensus_queue.cc:260] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:51.903744 20702 ts_tablet_manager.cc:1398] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240506 15:39:51.904196 20687 ts_tablet_manager.cc:1888] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:39:51.904211 20689 ts_tablet_manager.cc:1888] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:39:51.904213 20688 ts_tablet_manager.cc:1888] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:39:51.904250 20687 log.cc:1196] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/e1d48d4ad6d8437ba7764a47e10347f8
I20240506 15:39:51.904264 20689 log.cc:1196] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/e1d48d4ad6d8437ba7764a47e10347f8
I20240506 15:39:51.904275 20688 log.cc:1196] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/e1d48d4ad6d8437ba7764a47e10347f8
I20240506 15:39:51.904482 20687 ts_tablet_manager.cc:1909] T e1d48d4ad6d8437ba7764a47e10347f8 P 4a652030bf4c4f3599ac22e3ecba4a55: Deleting consensus metadata
I20240506 15:39:51.904505 20689 ts_tablet_manager.cc:1909] T e1d48d4ad6d8437ba7764a47e10347f8 P df44d35c45d14052bffed7d7411963ac: Deleting consensus metadata
I20240506 15:39:51.904523 20688 ts_tablet_manager.cc:1909] T e1d48d4ad6d8437ba7764a47e10347f8 P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting consensus metadata
I20240506 15:39:51.951166 20091 catalog_manager.cc:4830] TS ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207): tablet e1d48d4ad6d8437ba7764a47e10347f8 (table test-table [id=febc1763b554427993a6425295d2afce]) successfully deleted
I20240506 15:39:51.952045 20092 catalog_manager.cc:4830] TS df44d35c45d14052bffed7d7411963ac (127.17.240.195:38927): tablet e1d48d4ad6d8437ba7764a47e10347f8 (table test-table [id=febc1763b554427993a6425295d2afce]) successfully deleted
I20240506 15:39:51.972844 20090 catalog_manager.cc:4830] TS 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247): tablet e1d48d4ad6d8437ba7764a47e10347f8 (table test-table [id=febc1763b554427993a6425295d2afce]) successfully deleted
I20240506 15:39:52.154479 20668 raft_consensus.cc:482] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:39:52.154626 20668 raft_consensus.cc:504] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:52.154834 20668 leader_election.cc:288] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207), df44d35c45d14052bffed7d7411963ac (127.17.240.195:38927)
I20240506 15:39:52.155109 20542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e1328f26e14b2eb814ae46d2b5ce8c" candidate_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df44d35c45d14052bffed7d7411963ac" is_pre_election: true
I20240506 15:39:52.155125 20401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e1328f26e14b2eb814ae46d2b5ce8c" candidate_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" is_pre_election: true
I20240506 15:39:52.155205 20542 raft_consensus.cc:2454] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a652030bf4c4f3599ac22e3ecba4a55 in term 0.
I20240506 15:39:52.155208 20401 raft_consensus.cc:2454] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a652030bf4c4f3599ac22e3ecba4a55 in term 0.
I20240506 15:39:52.155372 20195 leader_election.cc:302] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a652030bf4c4f3599ac22e3ecba4a55, ab59a5185c7b42ab9736d0e0c6b5f561; no voters:
I20240506 15:39:52.155480 20668 raft_consensus.cc:2781] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:39:52.155540 20668 raft_consensus.cc:482] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:39:52.155556 20668 raft_consensus.cc:3037] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:52.176026 20668 raft_consensus.cc:504] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:52.176267 20668 leader_election.cc:288] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [CANDIDATE]: Term 1 election: Requested vote from peers ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207), df44d35c45d14052bffed7d7411963ac (127.17.240.195:38927)
I20240506 15:39:52.176524 20401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e1328f26e14b2eb814ae46d2b5ce8c" candidate_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561"
I20240506 15:39:52.176658 20401 raft_consensus.cc:3037] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:52.177634 20401 raft_consensus.cc:2454] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a652030bf4c4f3599ac22e3ecba4a55 in term 1.
I20240506 15:39:52.177884 20195 leader_election.cc:302] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a652030bf4c4f3599ac22e3ecba4a55, ab59a5185c7b42ab9736d0e0c6b5f561; no voters:
I20240506 15:39:52.177999 20668 raft_consensus.cc:2781] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:52.178066 20668 raft_consensus.cc:686] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [term 1 LEADER]: Becoming Leader. State: Replica: 4a652030bf4c4f3599ac22e3ecba4a55, State: Running, Role: LEADER
I20240506 15:39:52.178124 20668 consensus_queue.cc:237] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } }
I20240506 15:39:52.178814 20105 catalog_manager.cc:5484] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 45247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 } health_report { overall_health: UNKNOWN } } }
I20240506 15:39:52.181058 20542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e1328f26e14b2eb814ae46d2b5ce8c" candidate_uuid: "4a652030bf4c4f3599ac22e3ecba4a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df44d35c45d14052bffed7d7411963ac"
I20240506 15:39:52.181149 20542 raft_consensus.cc:3037] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:52.202075 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:52.200116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.17.240.193:45247)
0506 15:39:52.200269 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:52.200270 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:52.200279 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:39:52.200297 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:52.200298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:52.200307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:52.200361 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:52.200612 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:52.201115 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:52.201658 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:52.201798 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:52.201818 (+ 20us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:52.201843 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:52.201893 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:52.201954 (+ 61us) server_negotiation.cc:299] Negotiation successful
0506 15:39:52.201978 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20240506 15:39:52.206311 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:52.204445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:59559 (local address 127.17.240.193:45247)
0506 15:39:52.204493 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:52.204495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:52.204568 (+ 73us) server_negotiation.cc:407] Connection header received
0506 15:39:52.204647 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:52.204648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:52.204664 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:52.204708 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:39:52.204946 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:52.205373 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:52.205895 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:52.206024 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:52.206040 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:39:52.206061 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:39:52.206106 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:52.206178 (+ 72us) server_negotiation.cc:299] Negotiation successful
0506 15:39:52.206199 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240506 15:39:52.206332 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:52.204374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:45247 (local address 127.17.240.193:59559)
0506 15:39:52.204503 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:52.204513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:52.204590 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:52.204713 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:52.204714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:52.204716 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:39:52.204887 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:52.204888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:52.205439 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:52.205439 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:52.205828 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:52.205829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:52.205891 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:52.205897 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:39:52.206113 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:39:52.206115 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:39:52.206171 (+ 56us) client_negotiation.cc:240] Negotiation successful
0506 15:39:52.206223 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20240506 15:39:52.208542 20401 raft_consensus.cc:1261] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Refusing update from remote peer 4a652030bf4c4f3599ac22e3ecba4a55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:52.208756 20616 consensus_queue.cc:1035] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab59a5185c7b42ab9736d0e0c6b5f561" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 15:39:52.213176 20666 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240506 15:39:52.279136 20542 raft_consensus.cc:2454] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a652030bf4c4f3599ac22e3ecba4a55 in term 1.
I20240506 15:39:52.279356 20537 raft_consensus.cc:1261] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Refusing update from remote peer 4a652030bf4c4f3599ac22e3ecba4a55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:39:52.279902 20616 consensus_queue.cc:1035] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df44d35c45d14052bffed7d7411963ac" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 38927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:39:53.735059 20104 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35216:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:53.735172 20104 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35216:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:39:53.736045 20104 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 06a46e4298684345afd5e06658b79e91: Sending DeleteTablet for 3 replicas of tablet 14e1328f26e14b2eb814ae46d2b5ce8c
I20240506 15:39:53.736292 20233 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e1328f26e14b2eb814ae46d2b5ce8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:53 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:57068
I20240506 15:39:53.736305 20382 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e1328f26e14b2eb814ae46d2b5ce8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:53 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:51878
I20240506 15:39:53.736382 20522 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e1328f26e14b2eb814ae46d2b5ce8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:39:53 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:37000
I20240506 15:39:53.736570 20743 tablet_replica.cc:330] T 14e1328f26e14b2eb814ae46d2b5ce8c P 4a652030bf4c4f3599ac22e3ecba4a55: stopping tablet replica
I20240506 15:39:53.736563 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 20177
I20240506 15:39:53.736496 20744 tablet_replica.cc:330] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561: stopping tablet replica
I20240506 15:39:53.736642 20744 raft_consensus.cc:2229] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:53.737267 20744 raft_consensus.cc:2258] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:53.737406 20745 tablet_replica.cc:330] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac: stopping tablet replica
I20240506 15:39:53.737469 20745 raft_consensus.cc:2229] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:39:53.737510 20745 raft_consensus.cc:2258] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:39:53.737604 20744 ts_tablet_manager.cc:1875] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:53.737757 20745 ts_tablet_manager.cc:1875] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:39:53.739396 20744 ts_tablet_manager.cc:1888] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240506 15:39:53.739507 20744 log.cc:1196] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/14e1328f26e14b2eb814ae46d2b5ce8c
I20240506 15:39:53.739777 20745 ts_tablet_manager.cc:1888] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240506 15:39:53.739832 20744 ts_tablet_manager.cc:1909] T 14e1328f26e14b2eb814ae46d2b5ce8c P ab59a5185c7b42ab9736d0e0c6b5f561: Deleting consensus metadata
I20240506 15:39:53.739846 20745 log.cc:1196] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/14e1328f26e14b2eb814ae46d2b5ce8c
I20240506 15:39:53.740113 20745 ts_tablet_manager.cc:1909] T 14e1328f26e14b2eb814ae46d2b5ce8c P df44d35c45d14052bffed7d7411963ac: Deleting consensus metadata
I20240506 15:39:53.740149 20091 catalog_manager.cc:4830] TS ab59a5185c7b42ab9736d0e0c6b5f561 (127.17.240.194:44207): tablet 14e1328f26e14b2eb814ae46d2b5ce8c (table test-table [id=114813407295490dab556ae08828f679]) successfully deleted
I20240506 15:39:53.740588 20092 catalog_manager.cc:4830] TS df44d35c45d14052bffed7d7411963ac (127.17.240.195:38927): tablet 14e1328f26e14b2eb814ae46d2b5ce8c (table test-table [id=114813407295490dab556ae08828f679]) successfully deleted
W20240506 15:39:53.745184 20090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:45247 (error 108)
I20240506 15:39:53.745493 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 20317
I20240506 15:39:53.746380 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:53.745681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:45247 (local address 127.0.0.1:37752)
0506 15:39:53.746288 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:53.746338 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:45247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":66,"threads_started":1}
W20240506 15:39:53.747318 20090 catalog_manager.cc:4573] TS 4a652030bf4c4f3599ac22e3ecba4a55 (127.17.240.193:45247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14e1328f26e14b2eb814ae46d2b5ce8c: Network error: Client connection negotiation failed: client connection to 127.17.240.193:45247: connect: Connection refused (error 111)
I20240506 15:39:53.750646 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 20457
I20240506 15:39:53.757263 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 20073
2024-05-06T15:39:53Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5977 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:39:53 dist-test-slave-v0k1 krb5kdc[20750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:39:53 dist-test-slave-v0k1 krb5kdc[20750](info): set up 2 sockets
May 06 15:39:53 dist-test-slave-v0k1 krb5kdc[20750](info): commencing operation
krb5kdc: starting...
W20240506 15:39:54.718948 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:39:54 dist-test-slave-v0k1 krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:39:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:39:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:39:54.841898 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36051
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:36051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:54.921576 20763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:54.921783 20763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:54.921818 20763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:54.923158 20763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:39:54.923200 20763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:54.923223 20763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:54.923243 20763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:39:54.923262 20763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:39:54.924814 20763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:36051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:54.925035 20763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:54.925220 20763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 15:39:54.930464 20763 server_base.cc:1034] running on GCE node
W20240506 15:39:54.931180 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:54.931810 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:54.932493 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:54.932777 20763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:54.933022 20763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:54.934150 20763 hybrid_clock.cc:648] HybridClock initialized: now 1715009994934135 us; error 36 us; skew 500 ppm
May 06 15:39:54 dist-test-slave-v0k1 krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009994, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:54.936604 20763 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:39:54.937012 20763 webserver.cc:460] Webserver started at http://127.17.240.254:44319/ using document root <none> and password file <none>
I20240506 15:39:54.937196 20763 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:54.937235 20763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:54.937341 20763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:54.938342 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "f08721efc99a4202ab1bd1d8d2b40f5a"
format_stamp: "Formatted at 2024-05-06 15:39:54 on dist-test-slave-v0k1"
server_key: "aa17e27fc976e32397ec5f6a305698fa"
server_key_iv: "e1db43570e66032aef8422a637a81b30"
server_key_version: "encryptionkey@0"
I20240506 15:39:54.938644 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "f08721efc99a4202ab1bd1d8d2b40f5a"
format_stamp: "Formatted at 2024-05-06 15:39:54 on dist-test-slave-v0k1"
server_key: "aa17e27fc976e32397ec5f6a305698fa"
server_key_iv: "e1db43570e66032aef8422a637a81b30"
server_key_version: "encryptionkey@0"
I20240506 15:39:54.939872 20763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:39:54.940651 20778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:54.940850 20763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:54.940929 20763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "f08721efc99a4202ab1bd1d8d2b40f5a"
format_stamp: "Formatted at 2024-05-06 15:39:54 on dist-test-slave-v0k1"
server_key: "aa17e27fc976e32397ec5f6a305698fa"
server_key_iv: "e1db43570e66032aef8422a637a81b30"
server_key_version: "encryptionkey@0"
I20240506 15:39:54.940989 20763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:54.962947 20763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:54.964771 20763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:54.964895 20763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:54.968708 20763 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:36051
I20240506 15:39:54.968741 20830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:36051 every 8 connection(s)
I20240506 15:39:54.969063 20763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:39:54.969842 20831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:54.972052 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a: Bootstrap starting.
I20240506 15:39:54.972584 20831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:54.972805 20831 log.cc:824] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:54.973444 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a: No bootstrap required, opened a new log
I20240506 15:39:54.974841 20831 raft_consensus.cc:348] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08721efc99a4202ab1bd1d8d2b40f5a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36051 } }
I20240506 15:39:54.974972 20831 raft_consensus.cc:374] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:54.974988 20831 raft_consensus.cc:729] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08721efc99a4202ab1bd1d8d2b40f5a, State: Initialized, Role: FOLLOWER
I20240506 15:39:54.975143 20831 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08721efc99a4202ab1bd1d8d2b40f5a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36051 } }
I20240506 15:39:54.975224 20831 raft_consensus.cc:388] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:39:54.975246 20831 raft_consensus.cc:482] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:39:54.975272 20831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:54.975788 20831 raft_consensus.cc:504] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08721efc99a4202ab1bd1d8d2b40f5a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36051 } }
I20240506 15:39:54.975878 20831 leader_election.cc:302] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f08721efc99a4202ab1bd1d8d2b40f5a; no voters:
I20240506 15:39:54.976012 20831 leader_election.cc:288] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:39:54.976243 20831 sys_catalog.cc:564] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:39:54.976274 20834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:54.976449 20834 raft_consensus.cc:686] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [term 1 LEADER]: Becoming Leader. State: Replica: f08721efc99a4202ab1bd1d8d2b40f5a, State: Running, Role: LEADER
I20240506 15:39:54.976598 20834 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08721efc99a4202ab1bd1d8d2b40f5a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36051 } }
I20240506 15:39:54.976719 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 20763
I20240506 15:39:54.976840 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:39:54.977085 20835 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f08721efc99a4202ab1bd1d8d2b40f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08721efc99a4202ab1bd1d8d2b40f5a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36051 } } }
I20240506 15:39:54.977164 18371 external_mini_cluster.cc:1428] Setting key 803dc855e35cc909bdc675401a7cb2d0
I20240506 15:39:54.977157 20836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08721efc99a4202ab1bd1d8d2b40f5a. Latest consensus state: current_term: 1 leader_uuid: "f08721efc99a4202ab1bd1d8d2b40f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08721efc99a4202ab1bd1d8d2b40f5a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36051 } } }
I20240506 15:39:54.977281 20835 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:54.977613 20847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:39:54.977298 20836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:54.978113 20847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:39:54.979559 20847 catalog_manager.cc:1347] Generated new cluster ID: 79b1a9eb9b484c09a9b4fca48d65ae3f
I20240506 15:39:54.979616 20847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:39:54 dist-test-slave-v0k1 krb5kdc[20750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:54.986467 20847 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:39:54.986990 20847 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:39:54.988941 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:54.977855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44492 (local address 127.17.240.254:36051)
0506 15:39:54.978037 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:54.978050 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:54.978072 (+ 22us) server_negotiation.cc:407] Connection header received
0506 15:39:54.978415 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:54.978422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:54.978481 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:54.978635 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:54.979611 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:54.980807 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:54.981601 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:54.981813 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:54.985048 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:54.985056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:54.985056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:54.985079 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:54.987507 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:54.987887 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:54.987890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:54.987890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:54.987954 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:54.988102 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:54.988102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:54.988103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:54.988320 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:54.988451 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:54.988568 (+ 117us) server_negotiation.cc:299] Negotiation successful
0506 15:39:54.988726 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20240506 15:39:54.997571 20847 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f08721efc99a4202ab1bd1d8d2b40f5a: Generated new TSK 0
I20240506 15:39:54.997807 20847 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:39:55.034235 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36051
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:55.120456 20857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:55.120666 20857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:55.120699 20857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:55.122104 20857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:55.122164 20857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:55.122193 20857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:39:55.124044 20857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:55.124279 20857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:55.124477 20857 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:55.130301 20863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:55.130893 20862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:55.131647 20865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:55.133931 20857 server_base.cc:1034] running on GCE node
I20240506 15:39:55.134241 20857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:55.134528 20857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:55.135741 20857 hybrid_clock.cc:648] HybridClock initialized: now 1715009995135714 us; error 48 us; skew 500 ppm
May 06 15:39:55 dist-test-slave-v0k1 krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009995, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:55.138513 20857 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:39:55.139067 20857 webserver.cc:460] Webserver started at http://127.17.240.193:39843/ using document root <none> and password file <none>
I20240506 15:39:55.139290 20857 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:55.139340 20857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:55.139461 20857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:55.140533 20857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "6e5e8fa25b01405bac5e36c8d9ee7d87"
format_stamp: "Formatted at 2024-05-06 15:39:55 on dist-test-slave-v0k1"
server_key: "2a3fdc018a72b2fcf976ae4be1f5bced"
server_key_iv: "2658c42866ec79a097ada6ca0740ff72"
server_key_version: "encryptionkey@0"
I20240506 15:39:55.140857 20857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "6e5e8fa25b01405bac5e36c8d9ee7d87"
format_stamp: "Formatted at 2024-05-06 15:39:55 on dist-test-slave-v0k1"
server_key: "2a3fdc018a72b2fcf976ae4be1f5bced"
server_key_iv: "2658c42866ec79a097ada6ca0740ff72"
server_key_version: "encryptionkey@0"
I20240506 15:39:55.142263 20857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:39:55.143102 20872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:55.143271 20857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:55.143373 20857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "6e5e8fa25b01405bac5e36c8d9ee7d87"
format_stamp: "Formatted at 2024-05-06 15:39:55 on dist-test-slave-v0k1"
server_key: "2a3fdc018a72b2fcf976ae4be1f5bced"
server_key_iv: "2658c42866ec79a097ada6ca0740ff72"
server_key_version: "encryptionkey@0"
I20240506 15:39:55.143443 20857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:55.155993 20857 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:55.158237 20857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:55.158349 20857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:55.158592 20857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:55.158924 20857 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:55.158960 20857 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:55.158984 20857 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:55.158996 20857 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:55.164716 20857 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:44175
I20240506 15:39:55.164779 20985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:44175 every 8 connection(s)
I20240506 15:39:55.165063 20857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
May 06 15:39:55 dist-test-slave-v0k1 krb5kdc[20750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009995, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:55.173831 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 20857
I20240506 15:39:55.173951 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:39:55.174168 18371 external_mini_cluster.cc:1428] Setting key 0015f62ba05898d6d35c8461cbdf96c7
I20240506 15:39:55.175905 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:55.165806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:48351 (local address 127.17.240.254:36051)
0506 15:39:55.165883 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:55.165885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:55.166024 (+ 139us) server_negotiation.cc:407] Connection header received
0506 15:39:55.166459 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:55.166460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:55.166470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:55.166536 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:55.167371 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.168137 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:55.169041 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.169217 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:55.172218 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:55.172224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:55.172224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:55.172234 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:55.174222 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:55.174737 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:55.174738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:55.174739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:55.174772 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:55.174998 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:55.174999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:55.175000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:55.175141 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:55.175214 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:55.175688 (+ 474us) server_negotiation.cc:299] Negotiation successful
0506 15:39:55.175808 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240506 15:39:55.176035 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:55.165770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36051 (local address 127.17.240.193:48351)
0506 15:39:55.165903 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:55.165929 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:55.166365 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:55.166615 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:55.166619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:55.166950 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:55.167263 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:55.167267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.168240 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:55.168241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:55.168950 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:55.168954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.169453 (+ 499us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:55.170148 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:55.170174 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:55.172086 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:55.174318 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:55.174325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:55.174329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:55.174632 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:55.174853 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:55.174854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:55.174854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:55.174935 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:55.175230 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:55.175236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:55.175489 (+ 253us) client_negotiation.cc:769] Sending connection context
0506 15:39:55.175664 (+ 175us) client_negotiation.cc:240] Negotiation successful
0506 15:39:55.175826 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20240506 15:39:55.177063 20986 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36051
I20240506 15:39:55.177206 20986 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:55.177451 20986 heartbeater.cc:511] Master 127.17.240.254:36051 requested a full tablet report, sending...
I20240506 15:39:55.178017 20795 ts_manager.cc:194] Registered new tserver with Master: 6e5e8fa25b01405bac5e36c8d9ee7d87 (127.17.240.193:44175)
I20240506 15:39:55.179055 20795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:48351
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:39:55.207433 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36051
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:55.287349 20991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:55.287565 20991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:55.287602 20991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:55.288972 20991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:55.289029 20991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:55.289069 20991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:39:55.290733 20991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:55.291003 20991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:55.291250 20991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:55.296375 20996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:55.297014 20997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:55.297672 20999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:55.298369 20991 server_base.cc:1034] running on GCE node
I20240506 15:39:55.298591 20991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:55.298828 20991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:55.299975 20991 hybrid_clock.cc:648] HybridClock initialized: now 1715009995299948 us; error 43 us; skew 500 ppm
May 06 15:39:55 dist-test-slave-v0k1 krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009995, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:55.302629 20991 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:39:55.303079 20991 webserver.cc:460] Webserver started at http://127.17.240.194:32873/ using document root <none> and password file <none>
I20240506 15:39:55.303269 20991 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:55.303308 20991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:55.303397 20991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:55.304492 20991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "440515b28f2f486cb476c7702063117c"
format_stamp: "Formatted at 2024-05-06 15:39:55 on dist-test-slave-v0k1"
server_key: "6e7529073c35513203c414f33f65d440"
server_key_iv: "d426507723a2e7fd9f56401be3a59267"
server_key_version: "encryptionkey@0"
I20240506 15:39:55.304787 20991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "440515b28f2f486cb476c7702063117c"
format_stamp: "Formatted at 2024-05-06 15:39:55 on dist-test-slave-v0k1"
server_key: "6e7529073c35513203c414f33f65d440"
server_key_iv: "d426507723a2e7fd9f56401be3a59267"
server_key_version: "encryptionkey@0"
I20240506 15:39:55.306041 20991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:39:55.306746 21006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:55.306936 20991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:39:55.307046 20991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "440515b28f2f486cb476c7702063117c"
format_stamp: "Formatted at 2024-05-06 15:39:55 on dist-test-slave-v0k1"
server_key: "6e7529073c35513203c414f33f65d440"
server_key_iv: "d426507723a2e7fd9f56401be3a59267"
server_key_version: "encryptionkey@0"
I20240506 15:39:55.307127 20991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:55.319438 20991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:55.321498 20991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:55.321604 20991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:55.321792 20991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:55.322094 20991 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:55.322134 20991 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:55.322160 20991 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:55.322192 20991 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:55.327764 20991 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:39629
I20240506 15:39:55.327812 21119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:39629 every 8 connection(s)
I20240506 15:39:55.328234 20991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:39:55.332479 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 20991
I20240506 15:39:55.332587 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:39:55.332751 18371 external_mini_cluster.cc:1428] Setting key 445f032d161f7b1829ee3ed9154ffe6a
May 06 15:39:55 dist-test-slave-v0k1 krb5kdc[20750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009995, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:55.337060 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:55.329061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:42757 (local address 127.17.240.254:36051)
0506 15:39:55.329137 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:55.329139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:55.329263 (+ 124us) server_negotiation.cc:407] Connection header received
0506 15:39:55.329676 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:55.329677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:55.329688 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:55.329745 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:55.330464 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.330961 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:55.331632 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.331771 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:55.334249 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:55.334252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:55.334253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:55.334259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:55.335866 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:55.336227 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:55.336229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:55.336231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:55.336255 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:55.336430 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:55.336431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:55.336431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:55.336527 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:55.336589 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:55.336863 (+ 274us) server_negotiation.cc:299] Negotiation successful
0506 15:39:55.336964 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240506 15:39:55.337167 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:55.328944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36051 (local address 127.17.240.194:42757)
0506 15:39:55.329085 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:55.329122 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:55.329592 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:55.329805 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:55.329808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:55.330103 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:55.330391 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:55.330395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.331147 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:55.331147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:55.331536 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:55.331537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.331881 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:55.332455 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:55.332472 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:55.334150 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:55.335941 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:55.335946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:55.335949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:55.336147 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:55.336313 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:55.336313 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:55.336314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:55.336374 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:55.336586 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:55.336591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:55.336732 (+ 141us) client_negotiation.cc:769] Sending connection context
0506 15:39:55.336836 (+ 104us) client_negotiation.cc:240] Negotiation successful
0506 15:39:55.336975 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20240506 15:39:55.337599 21120 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36051
I20240506 15:39:55.337695 21120 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:55.337900 21120 heartbeater.cc:511] Master 127.17.240.254:36051 requested a full tablet report, sending...
I20240506 15:39:55.338291 20795 ts_manager.cc:194] Registered new tserver with Master: 440515b28f2f486cb476c7702063117c (127.17.240.194:39629)
I20240506 15:39:55.338675 20795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:42757
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:39:55.359084 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36051
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:55.442939 21125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:55.443202 21125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:55.443230 21125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:55.444674 21125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:55.444728 21125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:55.444754 21125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:39:55.446457 21125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:55.446667 21125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:55.446833 21125 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:55.452283 21133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:55.452886 21130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:55.453657 21131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:55.454361 21125 server_base.cc:1034] running on GCE node
I20240506 15:39:55.454600 21125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:55.454836 21125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:55.455966 21125 hybrid_clock.cc:648] HybridClock initialized: now 1715009995455947 us; error 38 us; skew 500 ppm
May 06 15:39:55 dist-test-slave-v0k1 krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009995, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:55.458456 21125 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:39:55.458889 21125 webserver.cc:460] Webserver started at http://127.17.240.195:45759/ using document root <none> and password file <none>
I20240506 15:39:55.459098 21125 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:55.459141 21125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:55.459242 21125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:55.460253 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "17e8d7708e51436683be3e8b7ce538d0"
format_stamp: "Formatted at 2024-05-06 15:39:55 on dist-test-slave-v0k1"
server_key: "5413270b552b44f623a44786c5e4534a"
server_key_iv: "c6434e6c50f5e9a8936a4dba73b8a3fa"
server_key_version: "encryptionkey@0"
I20240506 15:39:55.460562 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "17e8d7708e51436683be3e8b7ce538d0"
format_stamp: "Formatted at 2024-05-06 15:39:55 on dist-test-slave-v0k1"
server_key: "5413270b552b44f623a44786c5e4534a"
server_key_iv: "c6434e6c50f5e9a8936a4dba73b8a3fa"
server_key_version: "encryptionkey@0"
I20240506 15:39:55.461794 21125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:39:55.462428 21140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:55.462704 21125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:55.462797 21125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "17e8d7708e51436683be3e8b7ce538d0"
format_stamp: "Formatted at 2024-05-06 15:39:55 on dist-test-slave-v0k1"
server_key: "5413270b552b44f623a44786c5e4534a"
server_key_iv: "c6434e6c50f5e9a8936a4dba73b8a3fa"
server_key_version: "encryptionkey@0"
I20240506 15:39:55.462872 21125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:55.478075 21125 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:55.480371 21125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:55.480507 21125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:55.480764 21125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:55.481076 21125 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:55.481109 21125 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:55.481148 21125 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:55.481177 21125 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:55.487113 21125 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:33225
I20240506 15:39:55.487150 21253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:33225 every 8 connection(s)
I20240506 15:39:55.487481 21125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
May 06 15:39:55 dist-test-slave-v0k1 krb5kdc[20750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009995, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:55.493993 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 21125
I20240506 15:39:55.494100 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:39:55.494282 18371 external_mini_cluster.cc:1428] Setting key 7e390d217f016edc098e6dacefce7960
I20240506 15:39:55.496603 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:55.488291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:47569 (local address 127.17.240.254:36051)
0506 15:39:55.488382 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:55.488384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:55.488501 (+ 117us) server_negotiation.cc:407] Connection header received
0506 15:39:55.489110 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:55.489111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:55.489121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:55.489193 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:55.489986 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.490472 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:55.491064 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.491194 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:55.493588 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:55.493594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:55.493594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:55.493602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:55.495536 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:55.495907 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:55.495908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:55.495908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:55.495943 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:55.496079 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:55.496080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:55.496080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:55.496178 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:55.496221 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:55.496429 (+ 208us) server_negotiation.cc:299] Negotiation successful
0506 15:39:55.496534 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20240506 15:39:55.496693 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:55.488181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36051 (local address 127.17.240.195:47569)
0506 15:39:55.488382 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:55.488417 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:55.488973 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:55.489266 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:55.489270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:55.489597 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:55.489903 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:55.489907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.490540 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:55.490541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:55.490953 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:55.490955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:55.491380 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:55.491947 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:55.491963 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:55.493447 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:55.495609 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:55.495616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:55.495618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:55.495838 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:55.495986 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:55.495986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:55.495987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:55.496036 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:55.496217 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:55.496220 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:55.496331 (+ 111us) client_negotiation.cc:769] Sending connection context
0506 15:39:55.496415 (+ 84us) client_negotiation.cc:240] Negotiation successful
0506 15:39:55.496526 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":75,"threads_started":1}
I20240506 15:39:55.497164 21254 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36051
I20240506 15:39:55.497287 21254 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:55.497530 21254 heartbeater.cc:511] Master 127.17.240.254:36051 requested a full tablet report, sending...
I20240506 15:39:55.497977 20795 ts_manager.cc:194] Registered new tserver with Master: 17e8d7708e51436683be3e8b7ce538d0 (127.17.240.195:33225)
I20240506 15:39:55.498409 20795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:47569
I20240506 15:39:55.506739 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240506 15:39:55.517382 20852 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:39:55.516499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.17.240.254:36051)
0506 15:39:55.516591 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:55.516593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:55.516677 (+ 84us) server_negotiation.cc:407] Connection header received
0506 15:39:55.516725 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:55.516726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:55.516738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:55.516788 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:55.517320 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44496: BlockingRecv error: recv got EOF from 127.0.0.1:44496 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20240506 15:39:55.518316 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 20857
I20240506 15:39:55.523262 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 20991
I20240506 15:39:55.528021 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 21125
I20240506 15:39:55.532509 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 20763
2024-05-06T15:39:55Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1773 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:39:55 dist-test-slave-v0k1 krb5kdc[21266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:39:55 dist-test-slave-v0k1 krb5kdc[21266](info): set up 2 sockets
May 06 15:39:55 dist-test-slave-v0k1 krb5kdc[21266](info): commencing operation
krb5kdc: starting...
W20240506 15:39:56.495131 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:39:56 dist-test-slave-v0k1 krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:39:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:39:56.615695 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39247
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:35343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:39247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:56.696616 21279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:56.696822 21279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:56.696863 21279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:56.698518 21279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:39:56.698586 21279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:56.698614 21279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:56.698640 21279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:39:56.698660 21279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:39:56.700624 21279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:39247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:56.700917 21279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:56.701112 21279 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:56.706033 21287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:56.706825 21284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:56.707533 21285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:56.708272 21279 server_base.cc:1034] running on GCE node
I20240506 15:39:56.708499 21279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:56.708819 21279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:56.709952 21279 hybrid_clock.cc:648] HybridClock initialized: now 1715009996709931 us; error 42 us; skew 500 ppm
May 06 15:39:56 dist-test-slave-v0k1 krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009996, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:56.712364 21279 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:39:56.712800 21279 webserver.cc:460] Webserver started at http://127.17.240.254:35375/ using document root <none> and password file <none>
I20240506 15:39:56.713001 21279 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:56.713047 21279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:56.713152 21279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:56.714144 21279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86"
format_stamp: "Formatted at 2024-05-06 15:39:56 on dist-test-slave-v0k1"
server_key: "1dadaa2d6892c7e317f1a9fea466dcb0"
server_key_iv: "98d14cf58926b6596587b7868a388581"
server_key_version: "encryptionkey@0"
I20240506 15:39:56.714457 21279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86"
format_stamp: "Formatted at 2024-05-06 15:39:56 on dist-test-slave-v0k1"
server_key: "1dadaa2d6892c7e317f1a9fea466dcb0"
server_key_iv: "98d14cf58926b6596587b7868a388581"
server_key_version: "encryptionkey@0"
I20240506 15:39:56.715764 21279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:39:56.716532 21294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:56.716748 21279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:39:56.716840 21279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86"
format_stamp: "Formatted at 2024-05-06 15:39:56 on dist-test-slave-v0k1"
server_key: "1dadaa2d6892c7e317f1a9fea466dcb0"
server_key_iv: "98d14cf58926b6596587b7868a388581"
server_key_version: "encryptionkey@0"
I20240506 15:39:56.716902 21279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:56.733757 21279 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:56.735843 21279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:56.735973 21279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:56.739675 21279 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:39247
I20240506 15:39:56.739702 21346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:39247 every 8 connection(s)
I20240506 15:39:56.740026 21279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:39:56.740227 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 21279
I20240506 15:39:56.740342 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:39:56.740536 18371 external_mini_cluster.cc:1428] Setting key 3787800742b8edc93ddb83d48e4cf69a
I20240506 15:39:56.740721 21347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:56.743966 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86: Bootstrap starting.
I20240506 15:39:56.744597 21347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:56.744863 21347 log.cc:824] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:56.745703 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86: No bootstrap required, opened a new log
May 06 15:39:56 dist-test-slave-v0k1 krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:56.747570 21347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39247 } }
I20240506 15:39:56.747690 21347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:56.747707 21347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b47d8a6c19f4a05bb5e2a09c3fbab86, State: Initialized, Role: FOLLOWER
I20240506 15:39:56.747785 21347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39247 } }
I20240506 15:39:56.747833 21347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:39:56.747853 21347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:39:56.747884 21347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:56.748378 21347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39247 } }
I20240506 15:39:56.748461 21347 leader_election.cc:302] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b47d8a6c19f4a05bb5e2a09c3fbab86; no voters:
I20240506 15:39:56.748584 21347 leader_election.cc:288] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:39:56.748656 21352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:56.748780 21347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:39:56.748804 21352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [term 1 LEADER]: Becoming Leader. State: Replica: 2b47d8a6c19f4a05bb5e2a09c3fbab86, State: Running, Role: LEADER
I20240506 15:39:56.748888 21352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39247 } }
I20240506 15:39:56.749224 21354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b47d8a6c19f4a05bb5e2a09c3fbab86. Latest consensus state: current_term: 1 leader_uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39247 } } }
I20240506 15:39:56.749328 21354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:56.749488 21353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b47d8a6c19f4a05bb5e2a09c3fbab86" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39247 } } }
I20240506 15:39:56.749557 21353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86 [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:56.749547 21361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:39:56.750169 21361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:39:56.751600 21361 catalog_manager.cc:1347] Generated new cluster ID: 81eabddab7994f9f9e1669395edc3be0
I20240506 15:39:56.751645 21361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:39:56.755962 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:56.741039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.17.240.254:39247)
0506 15:39:56.741186 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:56.741199 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:56.741218 (+ 19us) server_negotiation.cc:407] Connection header received
0506 15:39:56.741666 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:56.741671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:56.741712 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:56.741826 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:56.742796 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:56.743537 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:56.744332 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:56.744529 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:56.746918 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:56.746931 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:56.746931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:56.746950 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:56.754207 (+ 7257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:56.754594 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:56.754596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:56.754597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:56.754650 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:56.754791 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:56.754792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:56.754792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:56.755032 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:56.755159 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:56.755633 (+ 474us) server_negotiation.cc:299] Negotiation successful
0506 15:39:56.755763 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20240506 15:39:56.757020 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:39:56.767402 21361 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:39:56.768189 21361 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:39:56.773258 21361 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2b47d8a6c19f4a05bb5e2a09c3fbab86: Generated new TSK 0
I20240506 15:39:56.773458 21361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:39:56.777936 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:56.774236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.17.240.254:39247)
0506 15:39:56.774281 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:56.774283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:56.774361 (+ 78us) server_negotiation.cc:407] Connection header received
0506 15:39:56.774431 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:56.774432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:56.774442 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:56.774509 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:56.775052 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:56.775858 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:56.776441 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:56.776624 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:56.777085 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:56.777088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:56.777089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:39:56.777621 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:56.777640 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:39:56.777648 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:56.777713 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:56.777804 (+ 91us) server_negotiation.cc:299] Negotiation successful
0506 15:39:56.777839 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240506 15:39:56.780377 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 21279
2024-05-06T15:39:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1245 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:39:56 dist-test-slave-v0k1 krb5kdc[21380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:39:56 dist-test-slave-v0k1 krb5kdc[21380](info): set up 2 sockets
May 06 15:39:56 dist-test-slave-v0k1 krb5kdc[21380](info): commencing operation
krb5kdc: starting...
W20240506 15:39:57.735333 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:39:57 dist-test-slave-v0k1 krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:39:57.856110 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43531
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:43531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:57.937361 21393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:57.937597 21393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:57.937619 21393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:57.939038 21393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:39:57.939088 21393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:57.939102 21393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:57.939114 21393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:39:57.939126 21393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:39:57.940698 21393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:43531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:57.940922 21393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:57.941125 21393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:57.946364 21399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:57.946982 21401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:57.947800 21398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:57.948467 21393 server_base.cc:1034] running on GCE node
I20240506 15:39:57.948726 21393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:57.948988 21393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:57.950131 21393 hybrid_clock.cc:648] HybridClock initialized: now 1715009997950115 us; error 40 us; skew 500 ppm
May 06 15:39:57 dist-test-slave-v0k1 krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009997, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:57.952790 21393 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:39:57.953210 21393 webserver.cc:460] Webserver started at http://127.17.240.254:35753/ using document root <none> and password file <none>
I20240506 15:39:57.953399 21393 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:57.953446 21393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:57.953572 21393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:57.954607 21393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "67fc9616b6d94749b59b3b5c6b2c04ad"
format_stamp: "Formatted at 2024-05-06 15:39:57 on dist-test-slave-v0k1"
server_key: "a6c8c004eb71646e6f4a296961b5d2c2"
server_key_iv: "67e1a6c2192a2f4e7cfab7e3d9def42f"
server_key_version: "encryptionkey@0"
I20240506 15:39:57.954923 21393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "67fc9616b6d94749b59b3b5c6b2c04ad"
format_stamp: "Formatted at 2024-05-06 15:39:57 on dist-test-slave-v0k1"
server_key: "a6c8c004eb71646e6f4a296961b5d2c2"
server_key_iv: "67e1a6c2192a2f4e7cfab7e3d9def42f"
server_key_version: "encryptionkey@0"
I20240506 15:39:57.956204 21393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:39:57.956985 21408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:57.957172 21393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:39:57.957258 21393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "67fc9616b6d94749b59b3b5c6b2c04ad"
format_stamp: "Formatted at 2024-05-06 15:39:57 on dist-test-slave-v0k1"
server_key: "a6c8c004eb71646e6f4a296961b5d2c2"
server_key_iv: "67e1a6c2192a2f4e7cfab7e3d9def42f"
server_key_version: "encryptionkey@0"
I20240506 15:39:57.957322 21393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:57.979676 21393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:57.982122 21393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:57.982259 21393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:57.986076 21393 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:43531
I20240506 15:39:57.986163 21460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:43531 every 8 connection(s)
I20240506 15:39:57.986495 21393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:39:57.987594 21461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:57.990443 21461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad: Bootstrap starting.
I20240506 15:39:57.991117 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 21393
I20240506 15:39:57.991163 21461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:57.991232 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:39:57.991431 18371 external_mini_cluster.cc:1428] Setting key 8ce2ea2ec15b4e44456003434b9ff8e8
I20240506 15:39:57.991436 21461 log.cc:824] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:57.992412 21461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad: No bootstrap required, opened a new log
I20240506 15:39:57.994851 21461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fc9616b6d94749b59b3b5c6b2c04ad" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43531 } }
I20240506 15:39:57.995064 21461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:57.995096 21461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67fc9616b6d94749b59b3b5c6b2c04ad, State: Initialized, Role: FOLLOWER
I20240506 15:39:57.995213 21461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fc9616b6d94749b59b3b5c6b2c04ad" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43531 } }
I20240506 15:39:57.995288 21461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:39:57.995328 21461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:39:57.995383 21461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:57.996232 21461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fc9616b6d94749b59b3b5c6b2c04ad" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43531 } }
I20240506 15:39:57.996362 21461 leader_election.cc:302] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67fc9616b6d94749b59b3b5c6b2c04ad; no voters:
I20240506 15:39:57.996575 21461 leader_election.cc:288] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:39:57.996835 21461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:39:57.996865 21466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:57.997143 21466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [term 1 LEADER]: Becoming Leader. State: Replica: 67fc9616b6d94749b59b3b5c6b2c04ad, State: Running, Role: LEADER
I20240506 15:39:57.997277 21466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fc9616b6d94749b59b3b5c6b2c04ad" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43531 } }
I20240506 15:39:57.997978 21468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67fc9616b6d94749b59b3b5c6b2c04ad. Latest consensus state: current_term: 1 leader_uuid: "67fc9616b6d94749b59b3b5c6b2c04ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fc9616b6d94749b59b3b5c6b2c04ad" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43531 } } }
I20240506 15:39:57.998092 21468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:57.998263 21467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67fc9616b6d94749b59b3b5c6b2c04ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fc9616b6d94749b59b3b5c6b2c04ad" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43531 } } }
I20240506 15:39:57.998335 21467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad [sys.catalog]: This master's current role is: LEADER
W20240506 15:39:57.999219 21482 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:39:57.999275 21482 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:39:57.999872 21475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:39:58.000095 21475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:39:58.001646 21475 catalog_manager.cc:1347] Generated new cluster ID: af814b172d7b42e4968dcc1a0f47f2c9
I20240506 15:39:58.001703 21475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:39:57 dist-test-slave-v0k1 krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:58.005612 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:57.992146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.17.240.254:43531)
0506 15:39:57.995167 (+ 3021us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:57.995181 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:57.995210 (+ 29us) server_negotiation.cc:407] Connection header received
0506 15:39:57.995341 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:57.995348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:57.995404 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:57.995627 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:57.996768 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:57.998665 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:57.999658 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:57.999874 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:58.002482 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:58.002509 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:58.002510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:58.002529 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:58.004315 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:58.004641 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:58.004644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:58.004645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:58.004693 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:58.004833 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:58.004834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:58.004834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:58.005021 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:58.005157 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:58.005291 (+ 134us) server_negotiation.cc:299] Negotiation successful
0506 15:39:58.005432 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2955,"thread_start_us":97,"threads_started":1}
I20240506 15:39:58.035732 21475 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:39:58.036314 21475 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:39:58.047328 21475 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 67fc9616b6d94749b59b3b5c6b2c04ad: Generated new TSK 0
I20240506 15:39:58.047564 21475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:39:58.087690 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43531
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:58.210029 21487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:58.210273 21487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:58.210309 21487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:58.212481 21487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:58.212546 21487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:58.212585 21487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:39:58.214932 21487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:58.215225 21487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:58.215443 21487 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:58.221081 21492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:58.221709 21495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:58.222515 21493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:58.223289 21487 server_base.cc:1034] running on GCE node
I20240506 15:39:58.223511 21487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:58.223749 21487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:58.224913 21487 hybrid_clock.cc:648] HybridClock initialized: now 1715009998224890 us; error 41 us; skew 500 ppm
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009998, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:58.227689 21487 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:39:58.228408 21487 webserver.cc:460] Webserver started at http://127.17.240.193:37491/ using document root <none> and password file <none>
I20240506 15:39:58.228614 21487 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:58.228665 21487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:58.228786 21487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:58.229914 21487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "3e5a520bd76c4f3597710dc3fa7fb944"
format_stamp: "Formatted at 2024-05-06 15:39:58 on dist-test-slave-v0k1"
server_key: "9ca932ac68f280e577496ade192384b4"
server_key_iv: "c13203d269cbf71f6d7f2549dc36d130"
server_key_version: "encryptionkey@0"
I20240506 15:39:58.230250 21487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "3e5a520bd76c4f3597710dc3fa7fb944"
format_stamp: "Formatted at 2024-05-06 15:39:58 on dist-test-slave-v0k1"
server_key: "9ca932ac68f280e577496ade192384b4"
server_key_iv: "c13203d269cbf71f6d7f2549dc36d130"
server_key_version: "encryptionkey@0"
I20240506 15:39:58.231606 21487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:39:58.232391 21502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:58.232636 21487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:58.232723 21487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "3e5a520bd76c4f3597710dc3fa7fb944"
format_stamp: "Formatted at 2024-05-06 15:39:58 on dist-test-slave-v0k1"
server_key: "9ca932ac68f280e577496ade192384b4"
server_key_iv: "c13203d269cbf71f6d7f2549dc36d130"
server_key_version: "encryptionkey@0"
I20240506 15:39:58.232786 21487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:58.248685 21487 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:58.251088 21487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:58.251217 21487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:58.251430 21487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:58.251855 21487 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:58.251901 21487 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:58.251936 21487 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:58.251956 21487 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:58.257639 21487 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:41699
I20240506 15:39:58.257691 21615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:41699 every 8 connection(s)
I20240506 15:39:58.257992 21487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009998, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:58.264052 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 21487
I20240506 15:39:58.264180 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:39:58.264374 18371 external_mini_cluster.cc:1428] Setting key b683188642d8aacf5d6340f43309ae9e
I20240506 15:39:58.267410 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:58.258747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:37209 (local address 127.17.240.254:43531)
0506 15:39:58.258831 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:58.258833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:58.259109 (+ 276us) server_negotiation.cc:407] Connection header received
0506 15:39:58.259643 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:58.259645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:58.259666 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:58.259730 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:58.260842 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.261388 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:58.261948 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.262132 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:58.264651 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:58.264657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:58.264657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:58.264665 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:58.266142 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:58.266532 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:58.266533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:58.266534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:58.266557 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:58.266738 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:58.266738 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:58.266739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:58.266831 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:58.266889 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:58.267205 (+ 316us) server_negotiation.cc:299] Negotiation successful
0506 15:39:58.267317 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240506 15:39:58.267539 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:58.258727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43531 (local address 127.17.240.193:37209)
0506 15:39:58.258947 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:58.258977 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:58.259506 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:58.259789 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:58.259793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:58.260135 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:58.260741 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:58.260747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.261480 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:58.261482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:58.261868 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:58.261869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.262014 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:58.262732 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:58.262751 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:58.264551 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:58.266228 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:58.266235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:58.266238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:58.266461 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:58.266616 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:58.266619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:58.266620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:58.266683 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:58.266884 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:58.266890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:58.267056 (+ 166us) client_negotiation.cc:769] Sending connection context
0506 15:39:58.267179 (+ 123us) client_negotiation.cc:240] Negotiation successful
0506 15:39:58.267331 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20240506 15:39:58.268152 21616 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43531
I20240506 15:39:58.268257 21616 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:58.268469 21616 heartbeater.cc:511] Master 127.17.240.254:43531 requested a full tablet report, sending...
I20240506 15:39:58.269064 21425 ts_manager.cc:194] Registered new tserver with Master: 3e5a520bd76c4f3597710dc3fa7fb944 (127.17.240.193:41699)
I20240506 15:39:58.270172 21425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:37209
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:39:58.291227 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43531
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:58.374240 21621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:58.374467 21621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:58.374501 21621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:58.375936 21621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:58.375991 21621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:58.376020 21621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:39:58.377825 21621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:58.378064 21621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:58.378235 21621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:58.384903 21626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:58.385433 21629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:58.386514 21627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:58.387625 21621 server_base.cc:1034] running on GCE node
I20240506 15:39:58.387879 21621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:58.388242 21621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:58.389443 21621 hybrid_clock.cc:648] HybridClock initialized: now 1715009998389404 us; error 54 us; skew 500 ppm
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009998, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:58.392438 21621 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:39:58.392951 21621 webserver.cc:460] Webserver started at http://127.17.240.194:44167/ using document root <none> and password file <none>
I20240506 15:39:58.393172 21621 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:58.393232 21621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:58.393370 21621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:58.394425 21621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "f0402e89c69640aeb86d3918e946965b"
format_stamp: "Formatted at 2024-05-06 15:39:58 on dist-test-slave-v0k1"
server_key: "ff5b1530a6571b364067a2ae460392d3"
server_key_iv: "41ca6dffa086b6d2608769e7014f0866"
server_key_version: "encryptionkey@0"
I20240506 15:39:58.394757 21621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "f0402e89c69640aeb86d3918e946965b"
format_stamp: "Formatted at 2024-05-06 15:39:58 on dist-test-slave-v0k1"
server_key: "ff5b1530a6571b364067a2ae460392d3"
server_key_iv: "41ca6dffa086b6d2608769e7014f0866"
server_key_version: "encryptionkey@0"
I20240506 15:39:58.396423 21621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:39:58.397181 21636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:58.397437 21621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:39:58.397524 21621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "f0402e89c69640aeb86d3918e946965b"
format_stamp: "Formatted at 2024-05-06 15:39:58 on dist-test-slave-v0k1"
server_key: "ff5b1530a6571b364067a2ae460392d3"
server_key_iv: "41ca6dffa086b6d2608769e7014f0866"
server_key_version: "encryptionkey@0"
I20240506 15:39:58.397604 21621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:58.404987 21621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:58.407891 21621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:58.408033 21621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:58.408274 21621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:58.408610 21621 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:58.408648 21621 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:58.408684 21621 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:58.408704 21621 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:58.414532 21621 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:44541
I20240506 15:39:58.414568 21749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:44541 every 8 connection(s)
I20240506 15:39:58.414896 21621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:39:58.415923 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 21621
I20240506 15:39:58.416034 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:39:58.416253 18371 external_mini_cluster.cc:1428] Setting key d5713f1a8c7d311c6a4d88846c29b8f9
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009998, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:58.425712 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:58.415707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:54579 (local address 127.17.240.254:43531)
0506 15:39:58.415777 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:58.415779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:58.415947 (+ 168us) server_negotiation.cc:407] Connection header received
0506 15:39:58.416683 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:58.416684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:58.416694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:58.416747 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:58.418289 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.418808 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:58.419465 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.419725 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:58.422204 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:58.422209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:58.422209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:58.422217 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:58.423987 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:58.424595 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:58.424597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:58.424597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:58.424635 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:58.424896 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:58.424897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:58.424897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:58.425008 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:58.425064 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:58.425537 (+ 473us) server_negotiation.cc:299] Negotiation successful
0506 15:39:58.425639 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240506 15:39:58.426003 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:58.415613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43531 (local address 127.17.240.194:54579)
0506 15:39:58.415786 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:58.415824 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:58.416516 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:58.416822 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:58.416826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:58.417307 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:58.418150 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:58.418159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.418927 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:58.418928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:58.419368 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:58.419370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.419524 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:58.420158 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:58.420185 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:58.422074 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:58.424115 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:58.424122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:58.424126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:58.424479 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:58.424713 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:58.424713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:58.424714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:58.424816 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:58.425076 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:58.425082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:58.425330 (+ 248us) client_negotiation.cc:769] Sending connection context
0506 15:39:58.425533 (+ 203us) client_negotiation.cc:240] Negotiation successful
0506 15:39:58.425764 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
I20240506 15:39:58.426728 21750 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43531
I20240506 15:39:58.426848 21750 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:58.427124 21750 heartbeater.cc:511] Master 127.17.240.254:43531 requested a full tablet report, sending...
I20240506 15:39:58.427688 21425 ts_manager.cc:194] Registered new tserver with Master: f0402e89c69640aeb86d3918e946965b (127.17.240.194:44541)
I20240506 15:39:58.428099 21425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:54579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:39:58.447882 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43531
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:58.530440 21755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:58.530675 21755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:58.530712 21755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:58.532153 21755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:58.532218 21755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:58.532260 21755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:39:58.533840 21755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:58.534093 21755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:58.534297 21755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:58.539685 21761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:58.540361 21760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:58.541050 21763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:58.541842 21755 server_base.cc:1034] running on GCE node
I20240506 15:39:58.542076 21755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:58.542313 21755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:58.543470 21755 hybrid_clock.cc:648] HybridClock initialized: now 1715009998543446 us; error 42 us; skew 500 ppm
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009998, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:58.546165 21755 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:39:58.546677 21755 webserver.cc:460] Webserver started at http://127.17.240.195:40137/ using document root <none> and password file <none>
I20240506 15:39:58.546862 21755 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:58.546909 21755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:58.547075 21755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:58.548233 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "92337a7187f1452db506c04ce7e94f7b"
format_stamp: "Formatted at 2024-05-06 15:39:58 on dist-test-slave-v0k1"
server_key: "3d934792582b776156a4c9ecaa172994"
server_key_iv: "7f6928d622aa9233a654999127858722"
server_key_version: "encryptionkey@0"
I20240506 15:39:58.548556 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "92337a7187f1452db506c04ce7e94f7b"
format_stamp: "Formatted at 2024-05-06 15:39:58 on dist-test-slave-v0k1"
server_key: "3d934792582b776156a4c9ecaa172994"
server_key_iv: "7f6928d622aa9233a654999127858722"
server_key_version: "encryptionkey@0"
I20240506 15:39:58.549984 21755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:39:58.550765 21770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:58.550998 21755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:58.551119 21755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "92337a7187f1452db506c04ce7e94f7b"
format_stamp: "Formatted at 2024-05-06 15:39:58 on dist-test-slave-v0k1"
server_key: "3d934792582b776156a4c9ecaa172994"
server_key_iv: "7f6928d622aa9233a654999127858722"
server_key_version: "encryptionkey@0"
I20240506 15:39:58.551189 21755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:58.560878 21755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:58.562803 21755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:58.562922 21755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:58.563175 21755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:39:58.563488 21755 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:39:58.563522 21755 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:58.563556 21755 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:39:58.563599 21755 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:58.569250 21755 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:36917
I20240506 15:39:58.569293 21883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:36917 every 8 connection(s)
I20240506 15:39:58.569602 21755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:39:58.572945 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 21755
I20240506 15:39:58.573096 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:39:58.573287 18371 external_mini_cluster.cc:1428] Setting key 17b96db872015d4b7c8ee3c6803d03be
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009998, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:58.578644 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:58.570379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:39811 (local address 127.17.240.254:43531)
0506 15:39:58.570437 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:58.570439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:58.570662 (+ 223us) server_negotiation.cc:407] Connection header received
0506 15:39:58.571129 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:58.571131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:58.571143 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:58.571217 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:58.572397 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.572902 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:58.573647 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.573833 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:58.576038 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:58.576043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:58.576043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:58.576050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:58.577421 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:58.577822 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:58.577823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:58.577824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:58.577849 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:58.578009 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:58.578009 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:58.578010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:58.578101 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:58.578150 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:58.578484 (+ 334us) server_negotiation.cc:299] Negotiation successful
0506 15:39:58.578576 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:39:58.578718 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:58.570336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43531 (local address 127.17.240.195:39811)
0506 15:39:58.570504 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0506 15:39:58.570533 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:39:58.570986 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:39:58.571329 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:39:58.571333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:39:58.571715 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:39:58.572290 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:58.572295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.573014 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:39:58.573015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:39:58.573556 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:39:58.573558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.573716 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:58.574301 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:39:58.574317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:39:58.575945 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:39:58.577518 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:58.577523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:58.577526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:58.577746 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:58.577906 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:39:58.577907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:39:58.577907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:39:58.577964 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:39:58.578179 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:39:58.578184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:39:58.578311 (+ 127us) client_negotiation.cc:769] Sending connection context
0506 15:39:58.578417 (+ 106us) client_negotiation.cc:240] Negotiation successful
0506 15:39:58.578534 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20240506 15:39:58.579227 21884 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43531
I20240506 15:39:58.579319 21884 heartbeater.cc:464] Registering TS with master...
I20240506 15:39:58.579515 21884 heartbeater.cc:511] Master 127.17.240.254:43531 requested a full tablet report, sending...
I20240506 15:39:58.579944 21425 ts_manager.cc:194] Registered new tserver with Master: 92337a7187f1452db506c04ce7e94f7b (127.17.240.195:36917)
I20240506 15:39:58.580319 21425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:39811
I20240506 15:39:58.585237 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009998, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009998, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:58.607640 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:58.600573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.17.240.254:43531)
0506 15:39:58.600682 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:58.600684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:58.600694 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:39:58.600722 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:58.600723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:58.600733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:58.600867 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:58.601674 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.602190 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:58.602812 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:58.602928 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:58.605182 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:58.605188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:58.605188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:58.605196 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:58.606692 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:58.606969 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:58.606970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:58.606970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:58.606993 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:58.607160 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:58.607160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:58.607160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:58.607287 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:58.607364 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:58.607439 (+ 75us) server_negotiation.cc:299] Negotiation successful
0506 15:39:58.607506 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20240506 15:39:58.607899 21425 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51832
I20240506 15:39:58.608332 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 21487
I20240506 15:39:58.613440 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 21621
I20240506 15:39:58.617918 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 21755
I20240506 15:39:58.622274 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 21393
2024-05-06T15:39:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1844 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21896](info): set up 2 sockets
May 06 15:39:58 dist-test-slave-v0k1 krb5kdc[21896](info): commencing operation
krb5kdc: starting...
W20240506 15:39:59.579195 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:39:59 dist-test-slave-v0k1 krb5kdc[21896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:39:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:39:59.699868 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36199
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:36199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:59.779697 21909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:59.779893 21909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:59.779929 21909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:59.781225 21909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:39:59.781280 21909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:59.781303 21909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:59.781323 21909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:39:59.781343 21909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:39:59.782966 21909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:36199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:59.783242 21909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:59.783438 21909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:59.788251 21917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:59.789191 21914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:59.789994 21915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:59.790815 21909 server_base.cc:1034] running on GCE node
I20240506 15:39:59.791070 21909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:39:59.791325 21909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:39:59.792471 21909 hybrid_clock.cc:648] HybridClock initialized: now 1715009999792445 us; error 38 us; skew 500 ppm
May 06 15:39:59 dist-test-slave-v0k1 krb5kdc[21896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009999, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:39:59.795001 21909 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:39:59.795423 21909 webserver.cc:460] Webserver started at http://127.17.240.254:39335/ using document root <none> and password file <none>
I20240506 15:39:59.795596 21909 fs_manager.cc:359] Metadata directory not provided
I20240506 15:39:59.795632 21909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:39:59.795717 21909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:39:59.796679 21909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "4d223c2f4bc244ef872f90d40604531c"
format_stamp: "Formatted at 2024-05-06 15:39:59 on dist-test-slave-v0k1"
server_key: "5a023a65e68700ad0a3a042b6d06e257"
server_key_iv: "f3fe1d090138cdff88bc9c08cd8b07d8"
server_key_version: "encryptionkey@0"
I20240506 15:39:59.796952 21909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "4d223c2f4bc244ef872f90d40604531c"
format_stamp: "Formatted at 2024-05-06 15:39:59 on dist-test-slave-v0k1"
server_key: "5a023a65e68700ad0a3a042b6d06e257"
server_key_iv: "f3fe1d090138cdff88bc9c08cd8b07d8"
server_key_version: "encryptionkey@0"
I20240506 15:39:59.798096 21909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:39:59.798822 21924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:39:59.799063 21909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:39:59.799151 21909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "4d223c2f4bc244ef872f90d40604531c"
format_stamp: "Formatted at 2024-05-06 15:39:59 on dist-test-slave-v0k1"
server_key: "5a023a65e68700ad0a3a042b6d06e257"
server_key_iv: "f3fe1d090138cdff88bc9c08cd8b07d8"
server_key_version: "encryptionkey@0"
I20240506 15:39:59.799212 21909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:39:59.826088 21909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:39:59.827899 21909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:39:59.828011 21909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:39:59.831576 21909 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:36199
I20240506 15:39:59.831641 21976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:36199 every 8 connection(s)
I20240506 15:39:59.831925 21909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:39:59.832536 21977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:39:59.834834 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 21909
I20240506 15:39:59.834921 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:39:59.835038 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c: Bootstrap starting.
I20240506 15:39:59.835121 18371 external_mini_cluster.cc:1428] Setting key 7028104fccad2a8720102e01472cc87d
I20240506 15:39:59.835642 21977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c: Neither blocks nor log segments found. Creating new log.
I20240506 15:39:59.835935 21977 log.cc:824] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c: Log is configured to *not* fsync() on all Append() calls
I20240506 15:39:59.836619 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c: No bootstrap required, opened a new log
I20240506 15:39:59.838239 21977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d223c2f4bc244ef872f90d40604531c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36199 } }
I20240506 15:39:59.838366 21977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:39:59.838394 21977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d223c2f4bc244ef872f90d40604531c, State: Initialized, Role: FOLLOWER
I20240506 15:39:59.838495 21977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d223c2f4bc244ef872f90d40604531c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36199 } }
I20240506 15:39:59.838585 21977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:39:59.838629 21977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:39:59.838682 21977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:39:59.839344 21977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d223c2f4bc244ef872f90d40604531c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36199 } }
I20240506 15:39:59.839458 21977 leader_election.cc:302] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d223c2f4bc244ef872f90d40604531c; no voters:
I20240506 15:39:59.839622 21977 leader_election.cc:288] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:39:59.839679 21982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:39:59.839833 21982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [term 1 LEADER]: Becoming Leader. State: Replica: 4d223c2f4bc244ef872f90d40604531c, State: Running, Role: LEADER
I20240506 15:39:59.839870 21977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:39:59.841236 21982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d223c2f4bc244ef872f90d40604531c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36199 } }
May 06 15:39:59 dist-test-slave-v0k1 krb5kdc[21896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715009999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:39:59.841912 21983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d223c2f4bc244ef872f90d40604531c. Latest consensus state: current_term: 1 leader_uuid: "4d223c2f4bc244ef872f90d40604531c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d223c2f4bc244ef872f90d40604531c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36199 } } }
I20240506 15:39:59.841915 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d223c2f4bc244ef872f90d40604531c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d223c2f4bc244ef872f90d40604531c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36199 } } }
I20240506 15:39:59.842378 21983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [sys.catalog]: This master's current role is: LEADER
I20240506 15:39:59.842386 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c [sys.catalog]: This master's current role is: LEADER
W20240506 15:39:59.842357 21996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:39:59.842658 21996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:39:59.842942 22000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:39:59.843487 22000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:39:59.846094 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:39:59.835663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.17.240.254:36199)
0506 15:39:59.836009 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0506 15:39:59.836018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:39:59.836041 (+ 23us) server_negotiation.cc:407] Connection header received
0506 15:39:59.836599 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:39:59.836606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:39:59.836671 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:39:59.836815 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:39:59.837766 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:59.838930 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:39:59.839778 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:39:59.840048 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:39:59.842885 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:39:59.842893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:39:59.842894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:39:59.842915 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:39:59.844847 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:59.845158 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:59.845163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:59.845163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:59.845207 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:39:59.845356 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:39:59.845357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:39:59.845357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:39:59.845538 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:39:59.845702 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0506 15:39:59.845806 (+ 104us) server_negotiation.cc:299] Negotiation successful
0506 15:39:59.845936 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":117,"threads_started":1}
I20240506 15:39:59.847919 22000 catalog_manager.cc:1347] Generated new cluster ID: c2f27f8556a748fe82a0ad4b7ff3b900
I20240506 15:39:59.847966 22000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:39:59.856000 22000 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:39:59.856567 22000 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:39:59.861299 22000 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4d223c2f4bc244ef872f90d40604531c: Generated new TSK 0
I20240506 15:39:59.861496 22000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:39:59.893992 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36199
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:39:59.987893 22004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:39:59.988076 22004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:39:59.988109 22004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:39:59.989411 22004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:39:59.989459 22004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:39:59.989498 22004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:39:59.991809 22004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:39:59.992053 22004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:39:59.992235 22004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:39:59.997735 22010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:59.998307 22009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:39:59.999192 22012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:39:59.999874 22004 server_base.cc:1034] running on GCE node
I20240506 15:40:00.000083 22004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:00.000308 22004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:00.001472 22004 hybrid_clock.cc:648] HybridClock initialized: now 1715010000001459 us; error 31 us; skew 500 ppm
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[21896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010000, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:00.004222 22004 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:40:00.004698 22004 webserver.cc:460] Webserver started at http://127.17.240.193:35423/ using document root <none> and password file <none>
I20240506 15:40:00.004915 22004 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:00.004964 22004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:00.005084 22004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:00.006135 22004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "dfd0555c9cfd464bab0766b9241db72d"
format_stamp: "Formatted at 2024-05-06 15:40:00 on dist-test-slave-v0k1"
server_key: "2ed1cf9f4d691925e0347a2c0c3132de"
server_key_iv: "efce22c225b87713ec40ab55c4b71f81"
server_key_version: "encryptionkey@0"
I20240506 15:40:00.006469 22004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "dfd0555c9cfd464bab0766b9241db72d"
format_stamp: "Formatted at 2024-05-06 15:40:00 on dist-test-slave-v0k1"
server_key: "2ed1cf9f4d691925e0347a2c0c3132de"
server_key_iv: "efce22c225b87713ec40ab55c4b71f81"
server_key_version: "encryptionkey@0"
I20240506 15:40:00.007792 22004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:40:00.008488 22019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:00.008668 22004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:00.008749 22004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "dfd0555c9cfd464bab0766b9241db72d"
format_stamp: "Formatted at 2024-05-06 15:40:00 on dist-test-slave-v0k1"
server_key: "2ed1cf9f4d691925e0347a2c0c3132de"
server_key_iv: "efce22c225b87713ec40ab55c4b71f81"
server_key_version: "encryptionkey@0"
I20240506 15:40:00.008816 22004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:00.020463 22004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:00.022230 22004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:00.022332 22004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:00.022544 22004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:00.022843 22004 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:00.022878 22004 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:00.022902 22004 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:00.022912 22004 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:00.028306 22004 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:33297
I20240506 15:40:00.028357 22132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:33297 every 8 connection(s)
I20240506 15:40:00.028636 22004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:40:00.029462 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 22004
I20240506 15:40:00.029572 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:40:00.029747 18371 external_mini_cluster.cc:1428] Setting key 04fbe5b56743330fca1e5006261b18f4
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[21896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010000, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
I20240506 15:40:00.037878 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:00.029297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:44123 (local address 127.17.240.254:36199)
0506 15:40:00.029354 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:00.029356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:00.029546 (+ 190us) server_negotiation.cc:407] Connection header received
0506 15:40:00.030125 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:00.030126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:00.030137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:00.030206 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:00.031041 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.031568 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:00.032181 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.032381 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:00.034770 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:00.034776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:00.034776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:00.034786 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:00.036486 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:00.036919 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:00.036920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:00.036921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:00.036949 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:00.037139 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:00.037140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:00.037141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:00.037264 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:00.037338 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:00.037626 (+ 288us) server_negotiation.cc:299] Negotiation successful
0506 15:40:00.037748 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:40:00.037899 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:00.029283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36199 (local address 127.17.240.193:44123)
0506 15:40:00.029434 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:00.029459 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:00.030017 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:00.030255 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:00.030260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:00.030631 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:00.030923 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:00.030927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.031681 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:00.031682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:00.032104 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:00.032106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.032452 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:00.032976 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:00.032993 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:00.034667 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:00.036595 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:00.036601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:00.036604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:00.036838 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:00.037021 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:00.037021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:00.037022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:00.037078 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:00.037333 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:00.037337 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:00.037447 (+ 110us) client_negotiation.cc:769] Sending connection context
0506 15:40:00.037596 (+ 149us) client_negotiation.cc:240] Negotiation successful
0506 15:40:00.037741 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":97,"threads_started":1}
I20240506 15:40:00.038391 22133 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36199
I20240506 15:40:00.038479 22133 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:00.038674 22133 heartbeater.cc:511] Master 127.17.240.254:36199 requested a full tablet report, sending...
I20240506 15:40:00.039230 21941 ts_manager.cc:194] Registered new tserver with Master: dfd0555c9cfd464bab0766b9241db72d (127.17.240.193:33297)
I20240506 15:40:00.039940 21941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:44123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:40:00.056339 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36199
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:00.136541 22138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:00.136744 22138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:00.136777 22138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:00.138111 22138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:00.138160 22138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:00.138200 22138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:40:00.139850 22138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:00.140121 22138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:00.140312 22138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:00.145663 22146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:00.146217 22143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:00.147045 22144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:00.147845 22138 server_base.cc:1034] running on GCE node
I20240506 15:40:00.148051 22138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:00.148262 22138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:00.149403 22138 hybrid_clock.cc:648] HybridClock initialized: now 1715010000149360 us; error 60 us; skew 500 ppm
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[21896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010000, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:00.152034 22138 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:40:00.152455 22138 webserver.cc:460] Webserver started at http://127.17.240.194:45795/ using document root <none> and password file <none>
I20240506 15:40:00.152694 22138 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:00.152760 22138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:00.152873 22138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:00.153877 22138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "ef59896dafc74c67ab60886d08fb0dad"
format_stamp: "Formatted at 2024-05-06 15:40:00 on dist-test-slave-v0k1"
server_key: "6838e843d58daa2637bdb4d22f2b25a3"
server_key_iv: "8c25958c7311fb3215df13ffb5b89824"
server_key_version: "encryptionkey@0"
I20240506 15:40:00.154181 22138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "ef59896dafc74c67ab60886d08fb0dad"
format_stamp: "Formatted at 2024-05-06 15:40:00 on dist-test-slave-v0k1"
server_key: "6838e843d58daa2637bdb4d22f2b25a3"
server_key_iv: "8c25958c7311fb3215df13ffb5b89824"
server_key_version: "encryptionkey@0"
I20240506 15:40:00.155476 22138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:40:00.156210 22153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:00.156409 22138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:40:00.156487 22138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "ef59896dafc74c67ab60886d08fb0dad"
format_stamp: "Formatted at 2024-05-06 15:40:00 on dist-test-slave-v0k1"
server_key: "6838e843d58daa2637bdb4d22f2b25a3"
server_key_iv: "8c25958c7311fb3215df13ffb5b89824"
server_key_version: "encryptionkey@0"
I20240506 15:40:00.156551 22138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:00.166026 22138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:00.168279 22138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:00.168411 22138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:00.168627 22138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:00.168920 22138 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:00.168956 22138 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:00.168990 22138 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:00.169029 22138 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:00.174490 22138 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:42763
I20240506 15:40:00.174553 22266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:42763 every 8 connection(s)
I20240506 15:40:00.175060 22138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[21896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010000, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:00.180855 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 22138
I20240506 15:40:00.180964 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:40:00.181157 18371 external_mini_cluster.cc:1428] Setting key 4212c269ffa7800c1d979ef805010f89
I20240506 15:40:00.183908 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:00.176049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36199 (local address 127.17.240.194:34949)
0506 15:40:00.176191 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:00.176218 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:00.176658 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:00.176852 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:00.176855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:00.177161 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:00.177449 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:00.177453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.178082 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:00.178083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:00.178446 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:00.178446 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.178782 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:00.179383 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:00.179400 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:00.180754 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:00.182515 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:00.182521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:00.182525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:00.182838 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:00.183065 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:00.183066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:00.183067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:00.183157 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:00.183433 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:00.183437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:00.183551 (+ 114us) client_negotiation.cc:769] Sending connection context
0506 15:40:00.183643 (+ 92us) client_negotiation.cc:240] Negotiation successful
0506 15:40:00.183757 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":87,"threads_started":1}
I20240506 15:40:00.183930 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:00.176059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:34949 (local address 127.17.240.254:36199)
0506 15:40:00.176124 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:00.176127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:00.176300 (+ 173us) server_negotiation.cc:407] Connection header received
0506 15:40:00.176746 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:00.176747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:00.176756 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:00.176806 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:00.177531 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.178014 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:00.178507 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.178608 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:00.180869 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:00.180874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:00.180875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:00.180883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:00.182431 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:00.182932 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:00.182934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:00.182934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:00.182969 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:00.183223 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:00.183224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:00.183225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:00.183359 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:00.183443 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:00.183676 (+ 233us) server_negotiation.cc:299] Negotiation successful
0506 15:40:00.183789 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:40:00.184669 22267 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36199
I20240506 15:40:00.184790 22267 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:00.185038 22267 heartbeater.cc:511] Master 127.17.240.254:36199 requested a full tablet report, sending...
I20240506 15:40:00.185479 21941 ts_manager.cc:194] Registered new tserver with Master: ef59896dafc74c67ab60886d08fb0dad (127.17.240.194:42763)
I20240506 15:40:00.185853 21941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:34949
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:40:00.206707 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36199
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:00.287865 22272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:00.288064 22272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:00.288110 22272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:00.289427 22272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:00.289482 22272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:00.289530 22272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:40:00.291162 22272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:00.291421 22272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:00.291651 22272 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:00.296702 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:00.297313 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:00.298027 22277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:00.298776 22272 server_base.cc:1034] running on GCE node
I20240506 15:40:00.298949 22272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:00.299198 22272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:00.300345 22272 hybrid_clock.cc:648] HybridClock initialized: now 1715010000300322 us; error 42 us; skew 500 ppm
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[21896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010000, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:00.302778 22272 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:40:00.303269 22272 webserver.cc:460] Webserver started at http://127.17.240.195:33225/ using document root <none> and password file <none>
I20240506 15:40:00.303464 22272 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:00.303509 22272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:00.303635 22272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:00.304605 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "0c9b436e9615451985b59f77b17a12bf"
format_stamp: "Formatted at 2024-05-06 15:40:00 on dist-test-slave-v0k1"
server_key: "0aa53dc71bb546e91425b8b560abf789"
server_key_iv: "02cdc4dcd337f73384f324213fd4a562"
server_key_version: "encryptionkey@0"
I20240506 15:40:00.304903 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "0c9b436e9615451985b59f77b17a12bf"
format_stamp: "Formatted at 2024-05-06 15:40:00 on dist-test-slave-v0k1"
server_key: "0aa53dc71bb546e91425b8b560abf789"
server_key_iv: "02cdc4dcd337f73384f324213fd4a562"
server_key_version: "encryptionkey@0"
I20240506 15:40:00.306069 22272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:40:00.306725 22287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:00.306890 22272 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:00.306970 22272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "0c9b436e9615451985b59f77b17a12bf"
format_stamp: "Formatted at 2024-05-06 15:40:00 on dist-test-slave-v0k1"
server_key: "0aa53dc71bb546e91425b8b560abf789"
server_key_iv: "02cdc4dcd337f73384f324213fd4a562"
server_key_version: "encryptionkey@0"
I20240506 15:40:00.307063 22272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:00.333530 22272 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:00.335307 22272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:00.335431 22272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:00.335647 22272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:00.335949 22272 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:00.335983 22272 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:00.336019 22272 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:00.336050 22272 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:00.341635 22272 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:37391
I20240506 15:40:00.341673 22400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:37391 every 8 connection(s)
I20240506 15:40:00.342175 22272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[21896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010000, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:00.350018 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:00.342907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:44579 (local address 127.17.240.254:36199)
0506 15:40:00.342975 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:00.342977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:00.343175 (+ 198us) server_negotiation.cc:407] Connection header received
0506 15:40:00.343636 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:00.343637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:00.343647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:00.343693 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:00.344411 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.344898 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:00.345444 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.345574 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:00.347828 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:00.347832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:00.347833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:00.347840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:00.349078 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:00.349405 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:00.349406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:00.349406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:00.349422 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:00.349554 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:00.349555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:00.349555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:00.349637 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:00.349682 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:00.349885 (+ 203us) server_negotiation.cc:299] Negotiation successful
0506 15:40:00.349949 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:40:00.350134 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:00.342872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36199 (local address 127.17.240.195:44579)
0506 15:40:00.343065 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:00.343096 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:00.343537 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:00.343751 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:00.343754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:00.344066 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:00.344342 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:00.344346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.344970 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:00.344971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:00.345346 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:00.345348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.345703 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:00.346292 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:00.346305 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:00.347733 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:00.349149 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:00.349154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:00.349156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:00.349348 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:00.349467 (+ 119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:00.349468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:00.349468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:00.349514 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:00.349679 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:00.349682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:00.349784 (+ 102us) client_negotiation.cc:769] Sending connection context
0506 15:40:00.349868 (+ 84us) client_negotiation.cc:240] Negotiation successful
0506 15:40:00.349971 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20240506 15:40:00.350610 22401 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36199
I20240506 15:40:00.350701 22401 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:00.350900 22401 heartbeater.cc:511] Master 127.17.240.254:36199 requested a full tablet report, sending...
I20240506 15:40:00.351294 21941 ts_manager.cc:194] Registered new tserver with Master: 0c9b436e9615451985b59f77b17a12bf (127.17.240.195:37391)
I20240506 15:40:00.351598 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 22272
I20240506 15:40:00.351683 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:40:00.351804 21941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:44579
I20240506 15:40:00.351891 18371 external_mini_cluster.cc:1428] Setting key 208f17ed319f6cc33e0f929f4a81dda3
I20240506 15:40:00.352645 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[21896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[21896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:40:00.373932 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:00.366951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.17.240.193:33297)
0506 15:40:00.367162 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:00.367169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:00.367181 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:40:00.367209 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:00.367211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:00.367224 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:00.367296 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:00.367964 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.368650 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:00.369191 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.369340 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:00.371391 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:00.371404 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:00.371404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:00.371411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:00.373087 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:00.373341 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:00.373343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:00.373344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:00.373370 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:00.373466 (+ 96us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:00.373467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:00.373467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:00.373561 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:00.373667 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:00.373758 (+ 91us) server_negotiation.cc:299] Negotiation successful
0506 15:40:00.373825 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":128,"threads_started":1}
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[21896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:00.381294 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:00.375210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.17.240.254:36199)
0506 15:40:00.375257 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:00.375259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:00.375287 (+ 28us) server_negotiation.cc:407] Connection header received
0506 15:40:00.375371 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:00.375372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:00.375381 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:00.375431 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:00.376012 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.376487 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:00.377005 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:00.377192 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:00.379141 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:00.379146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:00.379147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:00.379156 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:00.380320 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:00.380598 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:00.380599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:00.380600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:00.380623 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:00.380758 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:00.380759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:00.380759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:00.380881 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:00.380974 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:00.381083 (+ 109us) server_negotiation.cc:299] Negotiation successful
0506 15:40:00.381182 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20240506 15:40:00.381538 21941 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50408
I20240506 15:40:00.381911 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 22004
I20240506 15:40:00.386773 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 22138
I20240506 15:40:00.391434 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 22272
I20240506 15:40:00.395864 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 21909
2024-05-06T15:40:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1773 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[22413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[22413](info): set up 2 sockets
May 06 15:40:00 dist-test-slave-v0k1 krb5kdc[22413](info): commencing operation
krb5kdc: starting...
W20240506 15:40:01.355048 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:40:01 dist-test-slave-v0k1 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:40:01.474633 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:41801
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:41801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:01.554888 22426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:01.555130 22426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:01.555171 22426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:01.556676 22426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:01.556732 22426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:01.556757 22426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:01.556777 22426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:01.556794 22426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:40:01.558548 22426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:41801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:01.558826 22426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:01.559049 22426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:01.563889 22434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:01.564510 22432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:01.565372 22431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:01.566074 22426 server_base.cc:1034] running on GCE node
I20240506 15:40:01.566349 22426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:01.566625 22426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:01.567788 22426 hybrid_clock.cc:648] HybridClock initialized: now 1715010001567769 us; error 34 us; skew 500 ppm
May 06 15:40:01 dist-test-slave-v0k1 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010001, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:01.570436 22426 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:40:01.570784 22426 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:01.570837 22426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:01.570964 22426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:01.572055 22426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "77b6520ded4f4a6baacf9d11a27aeacd"
format_stamp: "Formatted at 2024-05-06 15:40:01 on dist-test-slave-v0k1"
server_key: "4afd6482a815c59eb972d1f699146368"
server_key_iv: "2d577d37d913545509e4f4fdc812bdd8"
server_key_version: "encryptionkey@0"
I20240506 15:40:01.572381 22426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "77b6520ded4f4a6baacf9d11a27aeacd"
format_stamp: "Formatted at 2024-05-06 15:40:01 on dist-test-slave-v0k1"
server_key: "4afd6482a815c59eb972d1f699146368"
server_key_iv: "2d577d37d913545509e4f4fdc812bdd8"
server_key_version: "encryptionkey@0"
I20240506 15:40:01.573673 22426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:40:01.574452 22440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:01.574726 22426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:01.574818 22426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "77b6520ded4f4a6baacf9d11a27aeacd"
format_stamp: "Formatted at 2024-05-06 15:40:01 on dist-test-slave-v0k1"
server_key: "4afd6482a815c59eb972d1f699146368"
server_key_iv: "2d577d37d913545509e4f4fdc812bdd8"
server_key_version: "encryptionkey@0"
I20240506 15:40:01.574893 22426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:01.586596 22426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:01.589164 22426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:01.589308 22426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:01.593431 22426 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:41801
I20240506 15:40:01.593502 22492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:41801 every 8 connection(s)
I20240506 15:40:01.593747 22426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:40:01.594389 22493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:01.597041 22493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd: Bootstrap starting.
I20240506 15:40:01.597637 22493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:01.597898 22493 log.cc:824] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:01.598660 22493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd: No bootstrap required, opened a new log
I20240506 15:40:01.599349 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 22426
I20240506 15:40:01.599465 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:40:01.599664 18371 external_mini_cluster.cc:1428] Setting key 60d74ea8823fefb49358fbdcb33e4942
I20240506 15:40:01.600785 22493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b6520ded4f4a6baacf9d11a27aeacd" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 41801 } }
I20240506 15:40:01.600925 22493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:01.600952 22493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b6520ded4f4a6baacf9d11a27aeacd, State: Initialized, Role: FOLLOWER
I20240506 15:40:01.601097 22493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b6520ded4f4a6baacf9d11a27aeacd" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 41801 } }
I20240506 15:40:01.601191 22493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:40:01.601227 22493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:40:01.601282 22493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:01.602083 22493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b6520ded4f4a6baacf9d11a27aeacd" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 41801 } }
I20240506 15:40:01.602190 22493 leader_election.cc:302] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77b6520ded4f4a6baacf9d11a27aeacd; no voters:
I20240506 15:40:01.602357 22493 leader_election.cc:288] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:40:01.602411 22498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:01.602530 22498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [term 1 LEADER]: Becoming Leader. State: Replica: 77b6520ded4f4a6baacf9d11a27aeacd, State: Running, Role: LEADER
I20240506 15:40:01.602612 22493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:01.602614 22498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b6520ded4f4a6baacf9d11a27aeacd" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 41801 } }
I20240506 15:40:01.603576 22500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77b6520ded4f4a6baacf9d11a27aeacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b6520ded4f4a6baacf9d11a27aeacd" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 41801 } } }
I20240506 15:40:01.603735 22500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:01.603734 22498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77b6520ded4f4a6baacf9d11a27aeacd. Latest consensus state: current_term: 1 leader_uuid: "77b6520ded4f4a6baacf9d11a27aeacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b6520ded4f4a6baacf9d11a27aeacd" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 41801 } } }
I20240506 15:40:01.603912 22498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:01.604166 22510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:40:01.604673 22510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
May 06 15:40:01 dist-test-slave-v0k1 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:01.607134 22510 catalog_manager.cc:1347] Generated new cluster ID: 025577557ea447829b41f4c9b37da576
I20240506 15:40:01.607192 22510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:40:01.610982 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:01.600234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.17.240.254:41801)
0506 15:40:01.600412 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:01.600422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:01.600443 (+ 21us) server_negotiation.cc:407] Connection header received
0506 15:40:01.600757 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:01.600762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:01.600811 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:01.600936 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:01.601926 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:01.603840 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:01.604685 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:01.604884 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:01.607369 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:01.607391 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:01.607392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:01.607413 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:01.609501 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:01.609920 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:01.609923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:01.609923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:01.609981 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:01.610142 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:01.610143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:01.610144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:01.610390 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:01.610522 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:01.610654 (+ 132us) server_negotiation.cc:299] Negotiation successful
0506 15:40:01.610780 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20240506 15:40:01.617213 22510 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:40:01.617930 22510 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:40:01.638513 22510 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd: Generated new TSK 0
I20240506 15:40:01.638746 22510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:40:01.670764 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:41801
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:01.751650 22519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:01.751842 22519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:01.751865 22519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:01.753216 22519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:01.753268 22519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:01.753295 22519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:40:01.754913 22519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:01.755146 22519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:01.755343 22519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:01.760674 22525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:01.761425 22524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:01.762282 22527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:01.763104 22519 server_base.cc:1034] running on GCE node
I20240506 15:40:01.763321 22519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:01.763566 22519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:01.764724 22519 hybrid_clock.cc:648] HybridClock initialized: now 1715010001764708 us; error 35 us; skew 500 ppm
May 06 15:40:01 dist-test-slave-v0k1 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010001, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:01.773618 22519 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:40:01.773969 22519 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:01.774014 22519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:01.774125 22519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:01.775239 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "80b282509931498685f83daac3f13991"
format_stamp: "Formatted at 2024-05-06 15:40:01 on dist-test-slave-v0k1"
server_key: "3fa96b946397b9902b784190175948c4"
server_key_iv: "a9326369650060be874b4fee1430da9b"
server_key_version: "encryptionkey@0"
I20240506 15:40:01.775552 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "80b282509931498685f83daac3f13991"
format_stamp: "Formatted at 2024-05-06 15:40:01 on dist-test-slave-v0k1"
server_key: "3fa96b946397b9902b784190175948c4"
server_key_iv: "a9326369650060be874b4fee1430da9b"
server_key_version: "encryptionkey@0"
I20240506 15:40:01.776769 22519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:40:01.777422 22533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:01.777637 22519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:01.777716 22519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "80b282509931498685f83daac3f13991"
format_stamp: "Formatted at 2024-05-06 15:40:01 on dist-test-slave-v0k1"
server_key: "3fa96b946397b9902b784190175948c4"
server_key_iv: "a9326369650060be874b4fee1430da9b"
server_key_version: "encryptionkey@0"
I20240506 15:40:01.777794 22519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:01.812186 22519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:01.814427 22519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:01.814528 22519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:01.814728 22519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:01.815088 22519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:01.815129 22519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:01.815162 22519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:01.815191 22519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:01.820729 22519 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:36015
I20240506 15:40:01.820789 22646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:36015 every 8 connection(s)
I20240506 15:40:01.821017 22519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:40:01.825678 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 22519
I20240506 15:40:01.825783 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:40:01.825963 18371 external_mini_cluster.cc:1428] Setting key 158341be49bd93ba01526bba3d7362ee
May 06 15:40:01 dist-test-slave-v0k1 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010001, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:01.829602 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:01.821633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:41801 (local address 127.17.240.193:34107)
0506 15:40:01.821805 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:01.821835 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:01.822312 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:01.822565 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:01.822568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:01.822870 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:01.823206 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:01.823210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:01.823896 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:01.823897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:01.824273 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:01.824275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:01.824708 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:01.825202 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:01.825215 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:01.826735 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:01.828416 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:01.828422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:01.828425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:01.828659 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:01.828835 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:01.828836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:01.828836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:01.828900 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:01.829149 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:01.829151 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:01.829263 (+ 112us) client_negotiation.cc:769] Sending connection context
0506 15:40:01.829349 (+ 86us) client_negotiation.cc:240] Negotiation successful
0506 15:40:01.829456 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20240506 15:40:01.829630 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:01.821700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:34107 (local address 127.17.240.254:41801)
0506 15:40:01.821763 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:01.821765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:01.821915 (+ 150us) server_negotiation.cc:407] Connection header received
0506 15:40:01.822414 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:01.822415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:01.822433 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:01.822499 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:01.823295 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:01.823810 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:01.824345 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:01.824535 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:01.826830 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:01.826832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:01.826833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:01.826838 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:01.828335 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:01.828728 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:01.828729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:01.828730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:01.828761 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:01.828962 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:01.828963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:01.828963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:01.829082 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:01.829160 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:01.829383 (+ 223us) server_negotiation.cc:299] Negotiation successful
0506 15:40:01.829489 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240506 15:40:01.830427 22647 heartbeater.cc:347] Connected to a master server at 127.17.240.254:41801
I20240506 15:40:01.830543 22647 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:01.830727 22647 heartbeater.cc:511] Master 127.17.240.254:41801 requested a full tablet report, sending...
I20240506 15:40:01.831395 22457 ts_manager.cc:194] Registered new tserver with Master: 80b282509931498685f83daac3f13991 (127.17.240.193:36015)
I20240506 15:40:01.832484 22457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:34107
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:40:01.854411 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:41801
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:01.934113 22652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:01.934300 22652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:01.934322 22652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:01.936074 22652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:01.936123 22652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:01.936148 22652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:40:01.937719 22652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:01.937902 22652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:01.938048 22652 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:01.943128 22657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:01.943667 22660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:01.944424 22658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:01.945180 22652 server_base.cc:1034] running on GCE node
I20240506 15:40:01.945348 22652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:01.945545 22652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:01.946681 22652 hybrid_clock.cc:648] HybridClock initialized: now 1715010001946662 us; error 36 us; skew 500 ppm
May 06 15:40:01 dist-test-slave-v0k1 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010001, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:01.949043 22652 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:40:01.949302 22652 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:01.949340 22652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:01.949420 22652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:01.950336 22652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "3915eb199ad44e3ab3e1e746475dc3ce"
format_stamp: "Formatted at 2024-05-06 15:40:01 on dist-test-slave-v0k1"
server_key: "fbfe660b8af32273a98069aa29033c89"
server_key_iv: "f22c991b3a7640dfb08443426272edcd"
server_key_version: "encryptionkey@0"
I20240506 15:40:01.950594 22652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "3915eb199ad44e3ab3e1e746475dc3ce"
format_stamp: "Formatted at 2024-05-06 15:40:01 on dist-test-slave-v0k1"
server_key: "fbfe660b8af32273a98069aa29033c89"
server_key_iv: "f22c991b3a7640dfb08443426272edcd"
server_key_version: "encryptionkey@0"
I20240506 15:40:01.951735 22652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240506 15:40:01.952343 22666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:01.952507 22652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:01.952575 22652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "3915eb199ad44e3ab3e1e746475dc3ce"
format_stamp: "Formatted at 2024-05-06 15:40:01 on dist-test-slave-v0k1"
server_key: "fbfe660b8af32273a98069aa29033c89"
server_key_iv: "f22c991b3a7640dfb08443426272edcd"
server_key_version: "encryptionkey@0"
I20240506 15:40:01.952623 22652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:01.966836 22652 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:01.968528 22652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:01.968623 22652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:01.968789 22652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:01.969067 22652 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:01.969099 22652 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:01.969125 22652 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:01.969152 22652 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:01.974220 22652 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:42391
I20240506 15:40:01.974288 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:42391 every 8 connection(s)
I20240506 15:40:01.974488 22652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:40:01.979116 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 22652
I20240506 15:40:01.979243 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:40:01.979451 18371 external_mini_cluster.cc:1428] Setting key d1d44c21a0d9085983aa4380032916a3
May 06 15:40:01 dist-test-slave-v0k1 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010001, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:01.983260 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:01.975161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:35629 (local address 127.17.240.254:41801)
0506 15:40:01.975207 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:01.975209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:01.975363 (+ 154us) server_negotiation.cc:407] Connection header received
0506 15:40:01.975840 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:01.975842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:01.975852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:01.975902 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:01.976595 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:01.977078 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:01.977622 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:01.977754 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:01.980216 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:01.980219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:01.980220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:01.980229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:01.981985 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:01.982414 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:01.982416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:01.982416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:01.982441 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:01.982639 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:01.982639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:01.982640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:01.982727 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:01.982775 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:01.983108 (+ 333us) server_negotiation.cc:299] Negotiation successful
0506 15:40:01.983192 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:40:01.983505 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:01.975145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:41801 (local address 127.17.240.194:35629)
0506 15:40:01.975274 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:01.975299 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:01.975750 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:01.975947 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:01.975950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:01.976245 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:01.976523 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:01.976527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:01.977140 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:01.977141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:01.977549 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:01.977551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:01.977873 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:01.978433 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:01.978453 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:01.980113 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:01.982067 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:01.982072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:01.982075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:01.982338 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:01.982502 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:01.982503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:01.982503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:01.982585 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:01.982778 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:01.982782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:01.982931 (+ 149us) client_negotiation.cc:769] Sending connection context
0506 15:40:01.983082 (+ 151us) client_negotiation.cc:240] Negotiation successful
0506 15:40:01.983252 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":76,"threads_started":1}
I20240506 15:40:01.984031 22780 heartbeater.cc:347] Connected to a master server at 127.17.240.254:41801
I20240506 15:40:01.984143 22780 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:01.984320 22780 heartbeater.cc:511] Master 127.17.240.254:41801 requested a full tablet report, sending...
I20240506 15:40:01.984751 22457 ts_manager.cc:194] Registered new tserver with Master: 3915eb199ad44e3ab3e1e746475dc3ce (127.17.240.194:42391)
I20240506 15:40:01.985117 22457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:35629
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:40:02.007725 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:41801
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:02.087134 22785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:02.087332 22785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:02.087354 22785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:02.088608 22785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:02.088647 22785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:02.088671 22785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:40:02.090176 22785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:02.090351 22785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:02.090502 22785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:02.095566 22791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:02.096279 22790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:02.097013 22793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:02.097783 22785 server_base.cc:1034] running on GCE node
I20240506 15:40:02.097967 22785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:02.098204 22785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:02.099350 22785 hybrid_clock.cc:648] HybridClock initialized: now 1715010002099336 us; error 29 us; skew 500 ppm
May 06 15:40:02 dist-test-slave-v0k1 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010002, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:02.101984 22785 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:40:02.102314 22785 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:02.102367 22785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:02.102514 22785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:02.103677 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "2bc7975561f24bd09993b06407b318e4"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "5e8cc0cbbb3ff5c1924e6994a544e423"
server_key_iv: "f5cae85c83a708a8aa23220b213cb589"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.104036 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "2bc7975561f24bd09993b06407b318e4"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "5e8cc0cbbb3ff5c1924e6994a544e423"
server_key_iv: "f5cae85c83a708a8aa23220b213cb589"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.105288 22785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:40:02.106014 22799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:02.106180 22785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:02.106257 22785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "2bc7975561f24bd09993b06407b318e4"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "5e8cc0cbbb3ff5c1924e6994a544e423"
server_key_iv: "f5cae85c83a708a8aa23220b213cb589"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.106323 22785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:02.133906 22785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:02.136032 22785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:02.136138 22785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:02.136360 22785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:02.136690 22785 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:02.136730 22785 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:02.136752 22785 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:02.136765 22785 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:02.142313 22785 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:35097
I20240506 15:40:02.142390 22912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:35097 every 8 connection(s)
I20240506 15:40:02.142604 22785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
May 06 15:40:02 dist-test-slave-v0k1 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010002, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:02.150748 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.143325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:58213 (local address 127.17.240.254:41801)
0506 15:40:02.143397 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.143400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.143568 (+ 168us) server_negotiation.cc:407] Connection header received
0506 15:40:02.144026 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.144027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.144037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.144103 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:02.144844 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.145319 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.145955 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.146059 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.148358 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:02.148362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:02.148363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:02.148369 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:02.149697 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:02.150055 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:02.150056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:02.150056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:02.150075 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:02.150219 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:02.150219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:02.150219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:02.150314 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:02.150364 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.150578 (+ 214us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.150675 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240506 15:40:02.150837 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.143287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:41801 (local address 127.17.240.195:58213)
0506 15:40:02.143454 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:02.143487 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:02.143934 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:02.144164 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:02.144167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:02.144479 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:02.144773 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.144777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.145419 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.145420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:02.145879 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.145881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.146263 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.146831 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:02.146849 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:02.148259 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:02.149783 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:02.149788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:02.149790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:02.149989 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:02.150125 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:02.150126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:02.150126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:02.150177 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:02.150360 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:02.150363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:02.150472 (+ 109us) client_negotiation.cc:769] Sending connection context
0506 15:40:02.150558 (+ 86us) client_negotiation.cc:240] Negotiation successful
0506 15:40:02.150669 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20240506 15:40:02.151458 22913 heartbeater.cc:347] Connected to a master server at 127.17.240.254:41801
I20240506 15:40:02.151619 22913 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:02.151773 22913 heartbeater.cc:511] Master 127.17.240.254:41801 requested a full tablet report, sending...
I20240506 15:40:02.152185 22457 ts_manager.cc:194] Registered new tserver with Master: 2bc7975561f24bd09993b06407b318e4 (127.17.240.195:35097)
I20240506 15:40:02.152482 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 22785
I20240506 15:40:02.152582 22457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:58213
I20240506 15:40:02.152604 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:40:02.152798 18371 external_mini_cluster.cc:1428] Setting key 74a6eae19115dfebb86443be8f6ece09
I20240506 15:40:02.153621 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:40:02.160107 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.154801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.17.240.254:41801)
0506 15:40:02.154840 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.154842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.154938 (+ 96us) server_negotiation.cc:407] Connection header received
0506 15:40:02.154993 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.154993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.155001 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.155082 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:02.155796 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.156251 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.156855 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.157102 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.158013 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:02.158017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:02.158017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:02.158022 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:02.159250 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:02.159573 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:02.159575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:02.159575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:02.159601 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:02.159723 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:02.159723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:02.159724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:02.159827 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:02.159904 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.159976 (+ 72us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.160043 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240506 15:40:02.161774 22457 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:40:02.163390 22457 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:40:02.168874 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.165703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:35097 (local address 127.0.0.1:34104)
0506 15:40:02.166168 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:02.166175 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:02.166217 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:02.166477 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:02.166478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:02.166481 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:02.166697 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.166699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.167735 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.167736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:02.168566 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.168568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.168644 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.168689 (+ 45us) client_negotiation.cc:769] Sending connection context
0506 15:40:02.168746 (+ 57us) client_negotiation.cc:240] Negotiation successful
0506 15:40:02.168794 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":87,"threads_started":1}
I20240506 15:40:02.169054 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.165703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:42391 (local address 127.0.0.1:56942)
0506 15:40:02.166274 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:02.166282 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:02.166331 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:02.166586 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:02.166587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:02.166589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:02.166770 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.166771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.168123 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.168124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:02.168847 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.168850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.168913 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.168917 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:40:02.168950 (+ 33us) client_negotiation.cc:240] Negotiation successful
0506 15:40:02.168987 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":10,"thread_start_us":84,"threads_started":1}
I20240506 15:40:02.169762 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.165655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:36015 (local address 127.0.0.1:60734)
0506 15:40:02.165966 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:02.165996 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:02.166083 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:02.166303 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:02.166306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:02.166309 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:02.166581 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.166584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.167915 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.167916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:02.168718 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.168720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.169601 (+ 881us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.169604 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:40:02.169658 (+ 54us) client_negotiation.cc:240] Negotiation successful
0506 15:40:02.169700 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":77,"threads_started":1}
I20240506 15:40:02.169888 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.165754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.17.240.193:36015)
0506 15:40:02.165925 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.165933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.166063 (+ 130us) server_negotiation.cc:407] Connection header received
0506 15:40:02.166181 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.166183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.166207 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.166283 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:02.166677 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.167838 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.168838 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.169425 (+ 587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.169481 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.169716 (+ 235us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.169774 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20240506 15:40:02.170058 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.165931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.17.240.195:35097)
0506 15:40:02.166088 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.166096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.166255 (+ 159us) server_negotiation.cc:407] Connection header received
0506 15:40:02.166302 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.166305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.166323 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.166394 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:02.166779 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.167649 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.169092 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.169702 (+ 610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.169752 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.169795 (+ 43us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.169994 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20240506 15:40:02.170318 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.165919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.17.240.194:42391)
0506 15:40:02.166137 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.166143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.166388 (+ 245us) server_negotiation.cc:407] Connection header received
0506 15:40:02.166427 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.166429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.166444 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.166512 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:02.166857 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.168046 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.168949 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.169735 (+ 786us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.169780 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.169810 (+ 30us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.170244 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20240506 15:40:02.170616 22581 tablet_service.cc:1467] Processing CreateTablet for tablet 281e6e89ea4544d0948927adb3a0ce07 (DEFAULT_TABLE table=test-table [id=a10282f0d5c64408b63da79cd50ffaf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:02.170624 22847 tablet_service.cc:1467] Processing CreateTablet for tablet 281e6e89ea4544d0948927adb3a0ce07 (DEFAULT_TABLE table=test-table [id=a10282f0d5c64408b63da79cd50ffaf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:02.170805 22714 tablet_service.cc:1467] Processing CreateTablet for tablet 281e6e89ea4544d0948927adb3a0ce07 (DEFAULT_TABLE table=test-table [id=a10282f0d5c64408b63da79cd50ffaf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:02.170936 22847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281e6e89ea4544d0948927adb3a0ce07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:02.170936 22581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281e6e89ea4544d0948927adb3a0ce07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:02.171113 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281e6e89ea4544d0948927adb3a0ce07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:02.173385 22930 tablet_bootstrap.cc:492] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: Bootstrap starting.
I20240506 15:40:02.173991 22931 tablet_bootstrap.cc:492] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: Bootstrap starting.
I20240506 15:40:02.174103 22930 tablet_bootstrap.cc:654] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:02.174248 22932 tablet_bootstrap.cc:492] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991: Bootstrap starting.
I20240506 15:40:02.174489 22930 log.cc:824] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:02.174571 22931 tablet_bootstrap.cc:654] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:02.174808 22932 tablet_bootstrap.cc:654] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:02.174839 22931 log.cc:824] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:02.175168 22932 log.cc:824] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:02.175431 22930 tablet_bootstrap.cc:492] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: No bootstrap required, opened a new log
I20240506 15:40:02.175487 22930 ts_tablet_manager.cc:1367] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:40:02.175837 22932 tablet_bootstrap.cc:492] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991: No bootstrap required, opened a new log
I20240506 15:40:02.175891 22932 ts_tablet_manager.cc:1367] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:40:02.177933 22930 raft_consensus.cc:348] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 } } peers { permanent_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 42391 } } peers { permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 } }
I20240506 15:40:02.178170 22930 raft_consensus.cc:374] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:02.178259 22930 raft_consensus.cc:729] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3915eb199ad44e3ab3e1e746475dc3ce, State: Initialized, Role: FOLLOWER
I20240506 15:40:02.178480 22930 consensus_queue.cc:260] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 } } peers { permanent_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 42391 } } peers { permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 } }
I20240506 15:40:02.178992 22930 ts_tablet_manager.cc:1398] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:40:02.178901 22932 raft_consensus.cc:348] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 } } peers { permanent_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 42391 } } peers { permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 } }
I20240506 15:40:02.179103 22780 heartbeater.cc:503] Master 127.17.240.254:41801 was elected leader, sending a full tablet report...
I20240506 15:40:02.179199 22932 raft_consensus.cc:374] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:02.179275 22932 raft_consensus.cc:729] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80b282509931498685f83daac3f13991, State: Initialized, Role: FOLLOWER
I20240506 15:40:02.179476 22932 consensus_queue.cc:260] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 } } peers { permanent_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 42391 } } peers { permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 } }
I20240506 15:40:02.180054 22931 tablet_bootstrap.cc:492] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: No bootstrap required, opened a new log
I20240506 15:40:02.180049 22932 ts_tablet_manager.cc:1398] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20240506 15:40:02.180135 22931 ts_tablet_manager.cc:1367] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240506 15:40:02.180182 22647 heartbeater.cc:503] Master 127.17.240.254:41801 was elected leader, sending a full tablet report...
I20240506 15:40:02.181979 22931 raft_consensus.cc:348] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 } } peers { permanent_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 42391 } } peers { permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 } }
I20240506 15:40:02.182089 22931 raft_consensus.cc:374] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:02.182116 22931 raft_consensus.cc:729] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc7975561f24bd09993b06407b318e4, State: Initialized, Role: FOLLOWER
I20240506 15:40:02.182199 22931 consensus_queue.cc:260] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 } } peers { permanent_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 42391 } } peers { permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 } }
I20240506 15:40:02.182439 22931 ts_tablet_manager.cc:1398] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:40:02.182494 22913 heartbeater.cc:503] Master 127.17.240.254:41801 was elected leader, sending a full tablet report...
W20240506 15:40:02.225014 22781 tablet.cc:2369] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:40:02.321792 22648 tablet.cc:2369] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:40:02.393144 22914 tablet.cc:2369] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:40:02.459460 22936 raft_consensus.cc:482] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:40:02.459597 22936 raft_consensus.cc:504] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 } } peers { permanent_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 42391 } } peers { permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 } }
I20240506 15:40:02.459949 22936 leader_election.cc:288] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80b282509931498685f83daac3f13991 (127.17.240.193:36015), 2bc7975561f24bd09993b06407b318e4 (127.17.240.195:35097)
I20240506 15:40:02.461970 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.460126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:36015 (local address 127.17.240.194:44639)
0506 15:40:02.460197 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:02.460225 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:02.460271 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:02.460398 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:02.460399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:02.460400 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:02.460576 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.460577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.461211 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.461212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:02.461758 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.461760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.461832 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.461835 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:40:02.461867 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:40:02.461900 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20240506 15:40:02.462155 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.460154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:35097 (local address 127.17.240.194:57167)
0506 15:40:02.460272 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:02.460282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:02.460344 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:02.460470 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:02.460471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:02.460473 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:02.460703 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.460705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.461353 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.461354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:02.461935 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:02.461936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.462005 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.462009 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:40:02.462041 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:40:02.462073 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20240506 15:40:02.462415 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.460237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:44639 (local address 127.17.240.193:36015)
0506 15:40:02.460280 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.460282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.460292 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:40:02.460324 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.460325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.460336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.460388 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:02.460642 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.461112 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.461839 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.462297 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.462306 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.462329 (+ 23us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.462364 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240506 15:40:02.462621 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.460200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:57167 (local address 127.17.240.195:35097)
0506 15:40:02.460250 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.460253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.460348 (+ 95us) server_negotiation.cc:407] Connection header received
0506 15:40:02.460397 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.460398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.460407 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.460461 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:02.460786 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.461251 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.462028 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.462496 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.462504 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.462530 (+ 26us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.462563 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240506 15:40:02.462692 22601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281e6e89ea4544d0948927adb3a0ce07" candidate_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80b282509931498685f83daac3f13991" is_pre_election: true
I20240506 15:40:02.462872 22601 raft_consensus.cc:2454] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3915eb199ad44e3ab3e1e746475dc3ce in term 0.
I20240506 15:40:02.462895 22867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281e6e89ea4544d0948927adb3a0ce07" candidate_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc7975561f24bd09993b06407b318e4" is_pre_election: true
I20240506 15:40:02.463035 22867 raft_consensus.cc:2454] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3915eb199ad44e3ab3e1e746475dc3ce in term 0.
I20240506 15:40:02.463192 22669 leader_election.cc:302] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3915eb199ad44e3ab3e1e746475dc3ce, 80b282509931498685f83daac3f13991; no voters:
I20240506 15:40:02.463349 22936 raft_consensus.cc:2781] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:40:02.463423 22936 raft_consensus.cc:482] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:40:02.463449 22936 raft_consensus.cc:3037] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:02.464087 22936 raft_consensus.cc:504] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 } } peers { permanent_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 42391 } } peers { permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 } }
I20240506 15:40:02.464213 22936 leader_election.cc:288] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [CANDIDATE]: Term 1 election: Requested vote from peers 80b282509931498685f83daac3f13991 (127.17.240.193:36015), 2bc7975561f24bd09993b06407b318e4 (127.17.240.195:35097)
I20240506 15:40:02.464407 22867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281e6e89ea4544d0948927adb3a0ce07" candidate_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc7975561f24bd09993b06407b318e4"
I20240506 15:40:02.464426 22601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281e6e89ea4544d0948927adb3a0ce07" candidate_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80b282509931498685f83daac3f13991"
I20240506 15:40:02.464495 22601 raft_consensus.cc:3037] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:02.464496 22867 raft_consensus.cc:3037] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:02.465365 22601 raft_consensus.cc:2454] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3915eb199ad44e3ab3e1e746475dc3ce in term 1.
I20240506 15:40:02.465361 22867 raft_consensus.cc:2454] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3915eb199ad44e3ab3e1e746475dc3ce in term 1.
I20240506 15:40:02.465559 22668 leader_election.cc:302] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc7975561f24bd09993b06407b318e4, 3915eb199ad44e3ab3e1e746475dc3ce; no voters:
I20240506 15:40:02.465658 22936 raft_consensus.cc:2781] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:02.465802 22936 raft_consensus.cc:686] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 1 LEADER]: Becoming Leader. State: Replica: 3915eb199ad44e3ab3e1e746475dc3ce, State: Running, Role: LEADER
I20240506 15:40:02.465909 22936 consensus_queue.cc:237] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 } } peers { permanent_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 42391 } } peers { permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 } }
I20240506 15:40:02.466720 22457 catalog_manager.cc:5484] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 3915eb199ad44e3ab3e1e746475dc3ce (127.17.240.194). New cstate: current_term: 1 leader_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3915eb199ad44e3ab3e1e746475dc3ce" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 42391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 } health_report { overall_health: UNKNOWN } } }
I20240506 15:40:02.525262 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.523302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.17.240.194:42391)
0506 15:40:02.523364 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.523366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.523374 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:40:02.523413 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.523414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.523431 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.523505 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:40:02.523719 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.524200 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:02.524812 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:02.524933 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:02.524970 (+ 37us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:40:02.525058 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:40:02.525121 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.525178 (+ 57us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.525203 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:40:02.527171 22867 raft_consensus.cc:1261] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4 [term 1 FOLLOWER]: Refusing update from remote peer 3915eb199ad44e3ab3e1e746475dc3ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:02.527171 22601 raft_consensus.cc:1261] T 281e6e89ea4544d0948927adb3a0ce07 P 80b282509931498685f83daac3f13991 [term 1 FOLLOWER]: Refusing update from remote peer 3915eb199ad44e3ab3e1e746475dc3ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:02.527529 22940 consensus_queue.cc:1035] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bc7975561f24bd09993b06407b318e4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:02.527657 22936 consensus_queue.cc:1035] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "80b282509931498685f83daac3f13991" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:02.530153 22945 mvcc.cc:205] Tried to move back new op lower bound from 7024680970348445696 to 7024680970100887552. Current Snapshot: MvccSnapshot[applied={T|T < 7024680970348445696}]
I20240506 15:40:02.530565 22944 mvcc.cc:205] Tried to move back new op lower bound from 7024680970348445696 to 7024680970100887552. Current Snapshot: MvccSnapshot[applied={T|T < 7024680970348445696}]
I20240506 15:40:02.532698 22457 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:02.532805 22457 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:02.533556 22457 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 77b6520ded4f4a6baacf9d11a27aeacd: Sending DeleteTablet for 3 replicas of tablet 281e6e89ea4544d0948927adb3a0ce07
I20240506 15:40:02.533890 22714 tablet_service.cc:1514] Processing DeleteTablet for tablet 281e6e89ea4544d0948927adb3a0ce07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:02 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:56942
I20240506 15:40:02.533926 22581 tablet_service.cc:1514] Processing DeleteTablet for tablet 281e6e89ea4544d0948927adb3a0ce07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:02 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:60734
I20240506 15:40:02.533924 22847 tablet_service.cc:1514] Processing DeleteTablet for tablet 281e6e89ea4544d0948927adb3a0ce07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:02 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:34104
I20240506 15:40:02.534233 22953 tablet_replica.cc:330] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: stopping tablet replica
I20240506 15:40:02.534334 22953 raft_consensus.cc:2229] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:40:02.534437 22953 raft_consensus.cc:2258] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:02.534458 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 22519
I20240506 15:40:02.535773 22953 ts_tablet_manager.cc:1875] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:02.537617 22953 ts_tablet_manager.cc:1888] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:02.537695 22953 log.cc:1196] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/281e6e89ea4544d0948927adb3a0ce07
I20240506 15:40:02.537954 22953 ts_tablet_manager.cc:1909] T 281e6e89ea4544d0948927adb3a0ce07 P 3915eb199ad44e3ab3e1e746475dc3ce: Deleting consensus metadata
I20240506 15:40:02.538398 22444 catalog_manager.cc:4830] TS 3915eb199ad44e3ab3e1e746475dc3ce (127.17.240.194:42391): tablet 281e6e89ea4544d0948927adb3a0ce07 (table test-table [id=a10282f0d5c64408b63da79cd50ffaf1]) successfully deleted
W20240506 15:40:02.542335 22443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:36015 (error 108)
I20240506 15:40:02.542436 22954 tablet_replica.cc:330] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: stopping tablet replica
I20240506 15:40:02.542563 22954 raft_consensus.cc:2229] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:02.542685 22954 raft_consensus.cc:2258] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:02.542686 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 22652
I20240506 15:40:02.543936 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.543502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:36015 (local address 127.0.0.1:60750)
0506 15:40:02.543835 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:02.543886 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308}
W20240506 15:40:02.544443 22443 catalog_manager.cc:4573] TS 80b282509931498685f83daac3f13991 (127.17.240.193:36015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 281e6e89ea4544d0948927adb3a0ce07: Network error: Client connection negotiation failed: client connection to 127.17.240.193:36015: connect: Connection refused (error 111)
I20240506 15:40:02.544998 22954 ts_tablet_manager.cc:1875] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:02.546667 22954 ts_tablet_manager.cc:1888] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:02.546741 22954 log.cc:1196] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/281e6e89ea4544d0948927adb3a0ce07
I20240506 15:40:02.546995 22954 ts_tablet_manager.cc:1909] T 281e6e89ea4544d0948927adb3a0ce07 P 2bc7975561f24bd09993b06407b318e4: Deleting consensus metadata
I20240506 15:40:02.547569 22442 catalog_manager.cc:4830] TS 2bc7975561f24bd09993b06407b318e4 (127.17.240.195:35097): tablet 281e6e89ea4544d0948927adb3a0ce07 (table test-table [id=a10282f0d5c64408b63da79cd50ffaf1]) successfully deleted
I20240506 15:40:02.547740 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 22785
I20240506 15:40:02.552908 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 22426
2024-05-06T15:40:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2157 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2024-05-06T15:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:02Z Disabled control of system clock
I20240506 15:40:02.580583 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36303
--webserver_interface=127.17.240.254
--webserver_port=0
--builtin_ntp_servers=127.17.240.212:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:36303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240506 15:40:02.661310 22960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:02.661496 22960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:02.661517 22960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:02.662848 22960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:02.662894 22960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:02.662906 22960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:02.662918 22960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:02.662930 22960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:40:02.664456 22960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:36303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:02.664680 22960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:02.664921 22960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:02.669724 22968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:02.670365 22966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:02.671147 22965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:02.671834 22960 server_base.cc:1034] running on GCE node
I20240506 15:40:02.672073 22960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:02.672310 22960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:02.673439 22960 hybrid_clock.cc:648] HybridClock initialized: now 1715010002673422 us; error 41 us; skew 500 ppm
I20240506 15:40:02.674770 22960 webserver.cc:460] Webserver started at http://127.17.240.254:33879/ using document root <none> and password file <none>
I20240506 15:40:02.674988 22960 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:02.675065 22960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:02.675194 22960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:02.676244 22960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "3b36180641ce441f9a24f28c93c4f204"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "cc330f81182c6bae44e1130cc35645cb"
server_key_iv: "24f136d0ac5d919edf471263bba8eac4"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.676554 22960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "3b36180641ce441f9a24f28c93c4f204"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "cc330f81182c6bae44e1130cc35645cb"
server_key_iv: "24f136d0ac5d919edf471263bba8eac4"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.677803 22960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:40:02.678474 22974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:02.678645 22960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:02.678723 22960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "3b36180641ce441f9a24f28c93c4f204"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "cc330f81182c6bae44e1130cc35645cb"
server_key_iv: "24f136d0ac5d919edf471263bba8eac4"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.678789 22960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:02.682366 22960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:02.682610 22960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:02.682714 22960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:02.686452 22960 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:36303
I20240506 15:40:02.686483 23026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:36303 every 8 connection(s)
I20240506 15:40:02.686796 22960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:40:02.687429 23027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:02.689965 23027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204: Bootstrap starting.
I20240506 15:40:02.690555 23027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:02.690810 23027 log.cc:824] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:02.691465 23027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204: No bootstrap required, opened a new log
I20240506 15:40:02.693101 23027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36180641ce441f9a24f28c93c4f204" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36303 } }
I20240506 15:40:02.693240 23027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:02.693259 23027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b36180641ce441f9a24f28c93c4f204, State: Initialized, Role: FOLLOWER
I20240506 15:40:02.693392 23027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36180641ce441f9a24f28c93c4f204" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36303 } }
I20240506 15:40:02.693472 23027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:40:02.693498 23027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:40:02.693527 23027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:02.694128 23027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36180641ce441f9a24f28c93c4f204" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36303 } }
I20240506 15:40:02.694229 23027 leader_election.cc:302] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b36180641ce441f9a24f28c93c4f204; no voters:
I20240506 15:40:02.694406 23027 leader_election.cc:288] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:40:02.694458 23030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:02.694595 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 22960
I20240506 15:40:02.694631 23030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [term 1 LEADER]: Becoming Leader. State: Replica: 3b36180641ce441f9a24f28c93c4f204, State: Running, Role: LEADER
I20240506 15:40:02.694691 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:40:02.694695 23027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:02.694726 23030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36180641ce441f9a24f28c93c4f204" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36303 } }
I20240506 15:40:02.694875 18371 external_mini_cluster.cc:1428] Setting key e61925ab320641846ecb3926e97c6fe1
I20240506 15:40:02.695075 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b36180641ce441f9a24f28c93c4f204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36180641ce441f9a24f28c93c4f204" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36303 } } }
I20240506 15:40:02.695148 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:02.695506 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b36180641ce441f9a24f28c93c4f204. Latest consensus state: current_term: 1 leader_uuid: "3b36180641ce441f9a24f28c93c4f204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36180641ce441f9a24f28c93c4f204" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36303 } } }
I20240506 15:40:02.695575 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:02.696084 23036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:40:02.696601 23036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:40:02.697903 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.695531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.17.240.254:36303)
0506 15:40:02.696208 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.696218 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.696243 (+ 25us) server_negotiation.cc:407] Connection header received
0506 15:40:02.696362 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.696368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.696409 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.696575 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:02.697372 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:02.697375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:02.697376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:02.697379 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:02.697406 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:02.697413 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:02.697491 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.697680 (+ 189us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.697729 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":70,"threads_started":1}
I20240506 15:40:02.698350 23036 catalog_manager.cc:1347] Generated new cluster ID: db521e7fd4964b839cefb9db67a01b66
I20240506 15:40:02.698506 23036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:40:02.710994 23036 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:40:02.711197 23036 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:40:02.719967 23036 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204: Generated new TSK 0
I20240506 15:40:02.720162 23036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:40:02.725165 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36303
--builtin_ntp_servers=127.17.240.212:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240506 15:40:02.804915 23051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:02.805136 23051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:02.805172 23051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:02.806545 23051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:02.806597 23051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:02.806639 23051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:40:02.808149 23051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:02.808406 23051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:02.808657 23051 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:02.814442 23056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:02.815040 23059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:02.815716 23057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:02.816499 23051 server_base.cc:1034] running on GCE node
I20240506 15:40:02.816681 23051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:02.816896 23051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:02.818064 23051 hybrid_clock.cc:648] HybridClock initialized: now 1715010002818040 us; error 42 us; skew 500 ppm
I20240506 15:40:02.819625 23051 webserver.cc:460] Webserver started at http://127.17.240.193:35203/ using document root <none> and password file <none>
I20240506 15:40:02.819854 23051 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:02.819902 23051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:02.820017 23051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:02.821028 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "9a2d68c5893f4e46b48c7802b8816cf1"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "6400d357900c70b3476b3c26a1f3166f"
server_key_iv: "204a77f23be649fd5a3e39a0d021f6ad"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.821348 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "9a2d68c5893f4e46b48c7802b8816cf1"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "6400d357900c70b3476b3c26a1f3166f"
server_key_iv: "204a77f23be649fd5a3e39a0d021f6ad"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.822573 23051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:40:02.823324 23065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:02.823485 23051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:02.823568 23051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "9a2d68c5893f4e46b48c7802b8816cf1"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "6400d357900c70b3476b3c26a1f3166f"
server_key_iv: "204a77f23be649fd5a3e39a0d021f6ad"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.823639 23051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:02.827215 23051 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:02.827455 23051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:02.827551 23051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:02.827749 23051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:02.828083 23051 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:02.828117 23051 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:02.828138 23051 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:02.828150 23051 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:02.834889 23051 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:46033
I20240506 15:40:02.834939 23178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:46033 every 8 connection(s)
I20240506 15:40:02.835295 23051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:40:02.837997 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.836064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:34319 (local address 127.17.240.254:36303)
0506 15:40:02.836123 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.836126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.836357 (+ 231us) server_negotiation.cc:407] Connection header received
0506 15:40:02.836840 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.836841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.836849 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.836914 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:02.837744 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:02.837745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:02.837745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:02.837749 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:02.837765 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:02.837768 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:02.837819 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.837901 (+ 82us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.837919 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240506 15:40:02.838241 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.836046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36303 (local address 127.17.240.193:34319)
0506 15:40:02.836247 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:02.836273 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:02.836732 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:02.836978 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:02.836981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:02.836989 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:02.837605 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:40:02.837605 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:02.837625 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:40:02.837649 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:02.837823 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:02.837828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:02.837830 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:40:02.837927 (+ 97us) client_negotiation.cc:240] Negotiation successful
0506 15:40:02.837980 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20240506 15:40:02.838812 23179 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36303
I20240506 15:40:02.838892 23179 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:02.839043 23179 heartbeater.cc:511] Master 127.17.240.254:36303 requested a full tablet report, sending...
I20240506 15:40:02.839535 22991 ts_manager.cc:194] Registered new tserver with Master: 9a2d68c5893f4e46b48c7802b8816cf1 (127.17.240.193:46033)
I20240506 15:40:02.839852 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 23051
I20240506 15:40:02.839968 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:40:02.840193 18371 external_mini_cluster.cc:1428] Setting key 4e2af97dba265a996d41160c8bd93c45
I20240506 15:40:02.841185 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36303
--builtin_ntp_servers=127.17.240.212:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240506 15:40:02.925007 23182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:02.925202 23182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:02.925225 23182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:02.926522 23182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:02.926560 23182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:02.926585 23182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:40:02.928086 23182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:02.928287 23182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:02.928503 23182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:02.933717 23188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:02.934245 23190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:02.934907 23187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:02.935810 23182 server_base.cc:1034] running on GCE node
I20240506 15:40:02.935972 23182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:02.936182 23182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:02.937330 23182 hybrid_clock.cc:648] HybridClock initialized: now 1715010002937316 us; error 30 us; skew 500 ppm
I20240506 15:40:02.938712 23182 webserver.cc:460] Webserver started at http://127.17.240.194:36187/ using document root <none> and password file <none>
I20240506 15:40:02.938913 23182 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:02.938961 23182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:02.939111 23182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:02.940155 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "17f6d337b63e4e4bafad3d314b697257"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "a6df81ffcb2002c100721e17d7d010ce"
server_key_iv: "e19a03f8e605f2c3b05481b144a8f5cf"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.940450 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "17f6d337b63e4e4bafad3d314b697257"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "a6df81ffcb2002c100721e17d7d010ce"
server_key_iv: "e19a03f8e605f2c3b05481b144a8f5cf"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.941666 23182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:40:02.942315 23196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:02.942476 23182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:40:02.942549 23182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "17f6d337b63e4e4bafad3d314b697257"
format_stamp: "Formatted at 2024-05-06 15:40:02 on dist-test-slave-v0k1"
server_key: "a6df81ffcb2002c100721e17d7d010ce"
server_key_iv: "e19a03f8e605f2c3b05481b144a8f5cf"
server_key_version: "encryptionkey@0"
I20240506 15:40:02.942610 23182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:02.946228 23182 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:02.946439 23182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:02.946545 23182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:02.946722 23182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:02.946995 23182 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:02.947064 23182 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:02.947090 23182 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:02.947117 23182 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:02.952356 23182 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:38545
I20240506 15:40:02.952427 23309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:38545 every 8 connection(s)
I20240506 15:40:02.952702 23182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:40:02.955438 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.953375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:37629 (local address 127.17.240.254:36303)
0506 15:40:02.953427 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:02.953430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:02.953649 (+ 219us) server_negotiation.cc:407] Connection header received
0506 15:40:02.954117 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:02.954118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:02.954124 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:02.954183 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:02.955092 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:02.955094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:02.955094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:02.955099 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:02.955139 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:02.955143 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:02.955207 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:02.955334 (+ 127us) server_negotiation.cc:299] Negotiation successful
0506 15:40:02.955350 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:40:02.955622 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:02.953345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36303 (local address 127.17.240.194:37629)
0506 15:40:02.953527 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:02.953551 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:02.954011 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:02.954248 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:02.954252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:02.954260 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:02.954895 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:40:02.954896 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:02.954928 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:40:02.954958 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:02.955208 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:02.955212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:02.955216 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:40:02.955347 (+ 131us) client_negotiation.cc:240] Negotiation successful
0506 15:40:02.955406 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20240506 15:40:02.956032 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 23182
I20240506 15:40:02.956135 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:40:02.956187 23310 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36303
I20240506 15:40:02.956285 23310 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:02.956319 18371 external_mini_cluster.cc:1428] Setting key 8cf5abd5e10a28eb2a58343dfdfa3ae4
I20240506 15:40:02.956421 23310 heartbeater.cc:511] Master 127.17.240.254:36303 requested a full tablet report, sending...
I20240506 15:40:02.956852 22990 ts_manager.cc:194] Registered new tserver with Master: 17f6d337b63e4e4bafad3d314b697257 (127.17.240.194:38545)
I20240506 15:40:02.957343 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36303
--builtin_ntp_servers=127.17.240.212:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240506 15:40:03.040310 23313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:03.040496 23313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:03.040526 23313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:03.041762 23313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:03.041802 23313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:03.041838 23313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:40:03.043295 23313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:03.043507 23313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:03.043743 23313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:03.048861 23319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:03.049407 23318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:03.050174 23321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:03.050904 23313 server_base.cc:1034] running on GCE node
I20240506 15:40:03.051102 23313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:03.051368 23313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:03.052518 23313 hybrid_clock.cc:648] HybridClock initialized: now 1715010003052504 us; error 28 us; skew 500 ppm
I20240506 15:40:03.053845 23313 webserver.cc:460] Webserver started at http://127.17.240.195:33113/ using document root <none> and password file <none>
I20240506 15:40:03.054044 23313 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:03.054092 23313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:03.054205 23313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:03.055253 23313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "7fbbbee6b6744800902a2ae0ef09c15d"
format_stamp: "Formatted at 2024-05-06 15:40:03 on dist-test-slave-v0k1"
server_key: "33465e8b55c157a917d6a2c2433bc988"
server_key_iv: "24ece6890680c43862c349bfb911509f"
server_key_version: "encryptionkey@0"
I20240506 15:40:03.055554 23313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "7fbbbee6b6744800902a2ae0ef09c15d"
format_stamp: "Formatted at 2024-05-06 15:40:03 on dist-test-slave-v0k1"
server_key: "33465e8b55c157a917d6a2c2433bc988"
server_key_iv: "24ece6890680c43862c349bfb911509f"
server_key_version: "encryptionkey@0"
I20240506 15:40:03.056706 23313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:03.057448 23327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:03.057621 23313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:03.057698 23313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "7fbbbee6b6744800902a2ae0ef09c15d"
format_stamp: "Formatted at 2024-05-06 15:40:03 on dist-test-slave-v0k1"
server_key: "33465e8b55c157a917d6a2c2433bc988"
server_key_iv: "24ece6890680c43862c349bfb911509f"
server_key_version: "encryptionkey@0"
I20240506 15:40:03.057754 23313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:03.061319 23313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:03.061513 23313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:03.061618 23313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:03.061780 23313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:03.062042 23313 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:03.062072 23313 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:03.062103 23313 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:03.062122 23313 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:03.067219 23313 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:39087
I20240506 15:40:03.067298 23440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:39087 every 8 connection(s)
I20240506 15:40:03.067552 23313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:40:03.069936 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.068265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:55213 (local address 127.17.240.254:36303)
0506 15:40:03.068327 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.068329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.068515 (+ 186us) server_negotiation.cc:407] Connection header received
0506 15:40:03.068958 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.068959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.068964 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.069050 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:03.069738 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:03.069739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:03.069739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:03.069743 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:03.069758 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:03.069761 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:03.069811 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:03.069867 (+ 56us) server_negotiation.cc:299] Negotiation successful
0506 15:40:03.069879 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:40:03.070178 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.068217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36303 (local address 127.17.240.195:55213)
0506 15:40:03.068420 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:03.068446 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:03.068866 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:03.069111 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:03.069114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:03.069122 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:03.069622 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:40:03.069622 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:03.069640 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:40:03.069661 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:03.069811 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:03.069813 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:03.069814 (+ 1us) client_negotiation.cc:769] Sending connection context
0506 15:40:03.069884 (+ 70us) client_negotiation.cc:240] Negotiation successful
0506 15:40:03.069921 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20240506 15:40:03.070803 23441 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36303
I20240506 15:40:03.070888 23441 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:03.071089 23441 heartbeater.cc:511] Master 127.17.240.254:36303 requested a full tablet report, sending...
I20240506 15:40:03.071492 22990 ts_manager.cc:194] Registered new tserver with Master: 7fbbbee6b6744800902a2ae0ef09c15d (127.17.240.195:39087)
I20240506 15:40:03.071722 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 23313
I20240506 15:40:03.071818 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:40:03.071985 18371 external_mini_cluster.cc:1428] Setting key 196c74a17feb7d833dfc88e86911e3a2
I20240506 15:40:03.072907 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:40:03.075420 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.074170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.17.240.254:36303)
0506 15:40:03.074246 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.074249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.074339 (+ 90us) server_negotiation.cc:407] Connection header received
0506 15:40:03.074411 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.074412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.074417 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.074467 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:03.075173 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:03.075174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:03.075175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:03.075179 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:03.075196 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:03.075200 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:03.075253 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:03.075329 (+ 76us) server_negotiation.cc:299] Negotiation successful
0506 15:40:03.075345 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240506 15:40:03.077080 22990 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:40:03.078500 22990 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:40:03.081977 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.080423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:39087 (local address 127.0.0.1:38916)
0506 15:40:03.080701 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:03.080713 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:03.080765 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:03.080999 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:03.081004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:03.081010 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:03.081545 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:40:03.081546 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:03.081576 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:40:03.081597 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:03.081801 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:03.081805 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:03.081807 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:40:03.081882 (+ 75us) client_negotiation.cc:240] Negotiation successful
0506 15:40:03.081896 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20240506 15:40:03.082008 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.080656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.17.240.195:39087)
0506 15:40:03.080826 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.080833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.080842 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:40:03.080868 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.080870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.080877 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.080978 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:03.081695 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:03.081701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:03.081701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:03.081705 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:03.081742 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:03.081748 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:03.081808 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:03.081893 (+ 85us) server_negotiation.cc:299] Negotiation successful
0506 15:40:03.081906 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20240506 15:40:03.082265 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.080426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:46033 (local address 127.0.0.1:34190)
0506 15:40:03.080838 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:03.080846 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:03.080877 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:03.081409 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:03.081410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:03.081412 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:03.081838 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:40:03.081839 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:03.081846 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:40:03.081857 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:03.082108 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:03.082115 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:03.082116 (+ 1us) client_negotiation.cc:769] Sending connection context
0506 15:40:03.082166 (+ 50us) client_negotiation.cc:240] Negotiation successful
0506 15:40:03.082181 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":108,"threads_started":1}
I20240506 15:40:03.082567 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.080819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.17.240.193:46033)
0506 15:40:03.081007 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.081014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.081023 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:40:03.081048 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.081050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.081060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.081136 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:03.081951 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:03.081957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:03.081958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:03.081963 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:03.082009 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:03.082016 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:03.082080 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:03.082434 (+ 354us) server_negotiation.cc:299] Negotiation successful
0506 15:40:03.082450 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20240506 15:40:03.082791 23375 tablet_service.cc:1467] Processing CreateTablet for tablet 2368c4b244fe4807ae92b1ea153d5b1b (DEFAULT_TABLE table=test-table [id=0ae4fd7ec0164668b56915144b06c105]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:03.083139 23375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2368c4b244fe4807ae92b1ea153d5b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:03.083112 23113 tablet_service.cc:1467] Processing CreateTablet for tablet 2368c4b244fe4807ae92b1ea153d5b1b (DEFAULT_TABLE table=test-table [id=0ae4fd7ec0164668b56915144b06c105]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:03.083281 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.080419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:38545 (local address 127.0.0.1:56432)
0506 15:40:03.080659 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:03.080680 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:03.080728 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:03.081813 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:03.081814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:03.081817 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:03.082506 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:40:03.082507 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:03.082515 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:40:03.082526 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:03.083129 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:03.083129 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:03.083130 (+ 1us) client_negotiation.cc:769] Sending connection context
0506 15:40:03.083187 (+ 57us) client_negotiation.cc:240] Negotiation successful
0506 15:40:03.083200 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":54,"threads_started":1}
I20240506 15:40:03.083314 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.080495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.17.240.194:38545)
0506 15:40:03.081008 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.081015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.081025 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:40:03.081049 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.081051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.081060 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.082600 (+ 1540us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:03.082992 (+ 392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:03.082994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:03.082994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:03.082997 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:03.083047 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:03.083053 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:03.083139 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:03.083199 (+ 60us) server_negotiation.cc:299] Negotiation successful
0506 15:40:03.083210 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":83,"threads_started":1}
I20240506 15:40:03.083374 23113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2368c4b244fe4807ae92b1ea153d5b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:03.083801 23244 tablet_service.cc:1467] Processing CreateTablet for tablet 2368c4b244fe4807ae92b1ea153d5b1b (DEFAULT_TABLE table=test-table [id=0ae4fd7ec0164668b56915144b06c105]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:03.083998 23244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2368c4b244fe4807ae92b1ea153d5b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:03.086367 23456 tablet_bootstrap.cc:492] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1: Bootstrap starting.
I20240506 15:40:03.086961 23456 tablet_bootstrap.cc:654] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:03.087152 23455 tablet_bootstrap.cc:492] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: Bootstrap starting.
I20240506 15:40:03.087260 23456 log.cc:824] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:03.087579 23455 tablet_bootstrap.cc:654] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:03.087576 23457 tablet_bootstrap.cc:492] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: Bootstrap starting.
I20240506 15:40:03.087827 23455 log.cc:824] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:03.088002 23456 tablet_bootstrap.cc:492] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1: No bootstrap required, opened a new log
I20240506 15:40:03.088078 23456 ts_tablet_manager.cc:1367] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:40:03.088145 23457 tablet_bootstrap.cc:654] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:03.088416 23457 log.cc:824] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:03.088485 23455 tablet_bootstrap.cc:492] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: No bootstrap required, opened a new log
I20240506 15:40:03.088551 23455 ts_tablet_manager.cc:1367] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:03.089087 23457 tablet_bootstrap.cc:492] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: No bootstrap required, opened a new log
I20240506 15:40:03.089156 23457 ts_tablet_manager.cc:1367] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:40:03.090294 23455 raft_consensus.cc:348] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f6d337b63e4e4bafad3d314b697257" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38545 } } peers { permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 } } peers { permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 } }
I20240506 15:40:03.090416 23455 raft_consensus.cc:374] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:03.090443 23455 raft_consensus.cc:729] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f6d337b63e4e4bafad3d314b697257, State: Initialized, Role: FOLLOWER
I20240506 15:40:03.090458 23456 raft_consensus.cc:348] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f6d337b63e4e4bafad3d314b697257" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38545 } } peers { permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 } } peers { permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 } }
I20240506 15:40:03.090600 23456 raft_consensus.cc:374] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:03.090626 23456 raft_consensus.cc:729] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2d68c5893f4e46b48c7802b8816cf1, State: Initialized, Role: FOLLOWER
I20240506 15:40:03.090565 23455 consensus_queue.cc:260] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f6d337b63e4e4bafad3d314b697257" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38545 } } peers { permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 } } peers { permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 } }
I20240506 15:40:03.090741 23456 consensus_queue.cc:260] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f6d337b63e4e4bafad3d314b697257" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38545 } } peers { permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 } } peers { permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 } }
I20240506 15:40:03.090824 23455 ts_tablet_manager.cc:1398] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:40:03.091051 23456 ts_tablet_manager.cc:1398] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:40:03.091089 23179 heartbeater.cc:503] Master 127.17.240.254:36303 was elected leader, sending a full tablet report...
I20240506 15:40:03.091331 23310 heartbeater.cc:503] Master 127.17.240.254:36303 was elected leader, sending a full tablet report...
I20240506 15:40:03.091434 23457 raft_consensus.cc:348] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f6d337b63e4e4bafad3d314b697257" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38545 } } peers { permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 } } peers { permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 } }
I20240506 15:40:03.091581 23457 raft_consensus.cc:374] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:03.091605 23457 raft_consensus.cc:729] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbbbee6b6744800902a2ae0ef09c15d, State: Initialized, Role: FOLLOWER
I20240506 15:40:03.091689 23457 consensus_queue.cc:260] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f6d337b63e4e4bafad3d314b697257" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38545 } } peers { permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 } } peers { permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 } }
I20240506 15:40:03.091917 23457 ts_tablet_manager.cc:1398] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240506 15:40:03.092031 23441 heartbeater.cc:503] Master 127.17.240.254:36303 was elected leader, sending a full tablet report...
I20240506 15:40:03.164001 23461 raft_consensus.cc:482] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:40:03.164145 23461 raft_consensus.cc:504] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f6d337b63e4e4bafad3d314b697257" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38545 } } peers { permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 } } peers { permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 } }
I20240506 15:40:03.164460 23461 leader_election.cc:288] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fbbbee6b6744800902a2ae0ef09c15d (127.17.240.195:39087), 9a2d68c5893f4e46b48c7802b8816cf1 (127.17.240.193:46033)
I20240506 15:40:03.165923 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.164747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:33159 (local address 127.17.240.195:39087)
0506 15:40:03.164802 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.164804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.164948 (+ 144us) server_negotiation.cc:407] Connection header received
0506 15:40:03.164979 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.164980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.165000 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.165049 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:03.165681 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:03.165682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:03.165682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:03.165687 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:03.165703 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:03.165708 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:03.165763 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:03.165826 (+ 63us) server_negotiation.cc:299] Negotiation successful
0506 15:40:03.165842 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240506 15:40:03.165956 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.164691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:39087 (local address 127.17.240.194:33159)
0506 15:40:03.164866 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:03.164874 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:03.164928 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:03.165079 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:03.165080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:03.165081 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:03.165577 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:40:03.165578 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:03.165586 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:40:03.165600 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:03.165771 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:03.165772 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:03.165773 (+ 1us) client_negotiation.cc:769] Sending connection context
0506 15:40:03.165833 (+ 60us) client_negotiation.cc:240] Negotiation successful
0506 15:40:03.165845 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":116,"threads_started":1}
I20240506 15:40:03.166152 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.164671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:46033 (local address 127.17.240.194:40067)
0506 15:40:03.164732 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:03.164744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:03.164791 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:03.165093 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:03.165094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:03.165096 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:03.165796 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:40:03.165797 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:03.165806 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:40:03.165817 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:03.166027 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:03.166028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:03.166029 (+ 1us) client_negotiation.cc:769] Sending connection context
0506 15:40:03.166074 (+ 45us) client_negotiation.cc:240] Negotiation successful
0506 15:40:03.166087 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20240506 15:40:03.166326 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2368c4b244fe4807ae92b1ea153d5b1b" candidate_uuid: "17f6d337b63e4e4bafad3d314b697257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" is_pre_election: true
I20240506 15:40:03.166393 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.164780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:40067 (local address 127.17.240.193:46033)
0506 15:40:03.164823 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.164825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.164834 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:40:03.164862 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.164863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.164889 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.164943 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:03.165910 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:03.165912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:03.165913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:03.165917 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:03.165935 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:03.165941 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:03.165999 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:03.166314 (+ 315us) server_negotiation.cc:299] Negotiation successful
0506 15:40:03.166335 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240506 15:40:03.166496 23395 raft_consensus.cc:2454] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f6d337b63e4e4bafad3d314b697257 in term 0.
I20240506 15:40:03.166675 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2368c4b244fe4807ae92b1ea153d5b1b" candidate_uuid: "17f6d337b63e4e4bafad3d314b697257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" is_pre_election: true
I20240506 15:40:03.166811 23133 raft_consensus.cc:2454] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f6d337b63e4e4bafad3d314b697257 in term 0.
I20240506 15:40:03.167037 23200 leader_election.cc:302] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f6d337b63e4e4bafad3d314b697257, 7fbbbee6b6744800902a2ae0ef09c15d; no voters:
I20240506 15:40:03.167222 23461 raft_consensus.cc:2781] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:40:03.167276 23461 raft_consensus.cc:482] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:40:03.167292 23461 raft_consensus.cc:3037] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:03.168094 23461 raft_consensus.cc:504] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f6d337b63e4e4bafad3d314b697257" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38545 } } peers { permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 } } peers { permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 } }
I20240506 15:40:03.168220 23461 leader_election.cc:288] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [CANDIDATE]: Term 1 election: Requested vote from peers 7fbbbee6b6744800902a2ae0ef09c15d (127.17.240.195:39087), 9a2d68c5893f4e46b48c7802b8816cf1 (127.17.240.193:46033)
I20240506 15:40:03.168443 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2368c4b244fe4807ae92b1ea153d5b1b" candidate_uuid: "17f6d337b63e4e4bafad3d314b697257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbbbee6b6744800902a2ae0ef09c15d"
I20240506 15:40:03.168520 23395 raft_consensus.cc:3037] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:03.169399 23395 raft_consensus.cc:2454] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f6d337b63e4e4bafad3d314b697257 in term 1.
I20240506 15:40:03.169551 23200 leader_election.cc:302] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f6d337b63e4e4bafad3d314b697257, 7fbbbee6b6744800902a2ae0ef09c15d; no voters:
I20240506 15:40:03.169656 23461 raft_consensus.cc:2781] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:03.169854 23461 raft_consensus.cc:686] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 1 LEADER]: Becoming Leader. State: Replica: 17f6d337b63e4e4bafad3d314b697257, State: Running, Role: LEADER
I20240506 15:40:03.169955 23461 consensus_queue.cc:237] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f6d337b63e4e4bafad3d314b697257" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38545 } } peers { permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 } } peers { permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 } }
I20240506 15:40:03.170773 22990 catalog_manager.cc:5484] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17f6d337b63e4e4bafad3d314b697257 (127.17.240.194). New cstate: current_term: 1 leader_uuid: "17f6d337b63e4e4bafad3d314b697257" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f6d337b63e4e4bafad3d314b697257" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 } health_report { overall_health: UNKNOWN } } }
I20240506 15:40:03.171291 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2368c4b244fe4807ae92b1ea153d5b1b" candidate_uuid: "17f6d337b63e4e4bafad3d314b697257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2d68c5893f4e46b48c7802b8816cf1"
I20240506 15:40:03.171396 23133 raft_consensus.cc:3037] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:03.172333 23133 raft_consensus.cc:2454] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f6d337b63e4e4bafad3d314b697257 in term 1.
I20240506 15:40:03.178615 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.177698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.17.240.194:38545)
0506 15:40:03.177753 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.177755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.177762 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:40:03.177799 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.177800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.177812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.177870 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:03.178425 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:03.178426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:03.178426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:03.178431 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:03.178449 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:03.178453 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:03.178514 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:03.178548 (+ 34us) server_negotiation.cc:299] Negotiation successful
0506 15:40:03.178561 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:40:03.180351 23133 raft_consensus.cc:1261] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1 [term 1 FOLLOWER]: Refusing update from remote peer 17f6d337b63e4e4bafad3d314b697257: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:03.180408 23395 raft_consensus.cc:1261] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d [term 1 FOLLOWER]: Refusing update from remote peer 17f6d337b63e4e4bafad3d314b697257: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:03.180694 23465 consensus_queue.cc:1035] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2d68c5893f4e46b48c7802b8816cf1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:03.180941 23465 consensus_queue.cc:1035] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fbbbee6b6744800902a2ae0ef09c15d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 39087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:03.183032 23468 mvcc.cc:205] Tried to move back new op lower bound from 7024680973023981568 to 7024680972984856576. Current Snapshot: MvccSnapshot[applied={T|T < 7024680973023981568}]
I20240506 15:40:03.183444 23469 mvcc.cc:205] Tried to move back new op lower bound from 7024680973023981568 to 7024680972984856576. Current Snapshot: MvccSnapshot[applied={T|T < 7024680973023981568 or (T in {7024680973023981568})}]
I20240506 15:40:03.185066 23471 mvcc.cc:205] Tried to move back new op lower bound from 7024680973023981568 to 7024680972984856576. Current Snapshot: MvccSnapshot[applied={T|T < 7024680973023981568 or (T in {7024680973023981568})}]
I20240506 15:40:03.185963 22989 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60340:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:03.186053 22989 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60340:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:03.186754 22989 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 3b36180641ce441f9a24f28c93c4f204: Sending DeleteTablet for 3 replicas of tablet 2368c4b244fe4807ae92b1ea153d5b1b
I20240506 15:40:03.187093 23375 tablet_service.cc:1514] Processing DeleteTablet for tablet 2368c4b244fe4807ae92b1ea153d5b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:03 UTC) from {username='slave'} at 127.0.0.1:38916
I20240506 15:40:03.187119 23113 tablet_service.cc:1514] Processing DeleteTablet for tablet 2368c4b244fe4807ae92b1ea153d5b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:03 UTC) from {username='slave'} at 127.0.0.1:34190
I20240506 15:40:03.187319 23244 tablet_service.cc:1514] Processing DeleteTablet for tablet 2368c4b244fe4807ae92b1ea153d5b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:03 UTC) from {username='slave'} at 127.0.0.1:56432
I20240506 15:40:03.187530 23479 tablet_replica.cc:330] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: stopping tablet replica
I20240506 15:40:03.187594 23478 tablet_replica.cc:330] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1: stopping tablet replica
I20240506 15:40:03.187641 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 23051
I20240506 15:40:03.187696 23478 raft_consensus.cc:2229] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:03.187781 23479 raft_consensus.cc:2229] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:40:03.187801 23478 raft_consensus.cc:2258] T 2368c4b244fe4807ae92b1ea153d5b1b P 9a2d68c5893f4e46b48c7802b8816cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:03.187940 23479 raft_consensus.cc:2258] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:03.187942 23477 tablet_replica.cc:330] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: stopping tablet replica
I20240506 15:40:03.188016 23477 raft_consensus.cc:2229] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:03.188102 23477 raft_consensus.cc:2258] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:03.188688 23479 ts_tablet_manager.cc:1875] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:03.188695 23477 ts_tablet_manager.cc:1875] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:03.190774 23477 ts_tablet_manager.cc:1888] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:03.190857 23477 log.cc:1196] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/2368c4b244fe4807ae92b1ea153d5b1b
I20240506 15:40:03.190805 23479 ts_tablet_manager.cc:1888] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:03.191110 23479 log.cc:1196] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/2368c4b244fe4807ae92b1ea153d5b1b
I20240506 15:40:03.191197 23477 ts_tablet_manager.cc:1909] T 2368c4b244fe4807ae92b1ea153d5b1b P 7fbbbee6b6744800902a2ae0ef09c15d: Deleting consensus metadata
I20240506 15:40:03.191519 23479 ts_tablet_manager.cc:1909] T 2368c4b244fe4807ae92b1ea153d5b1b P 17f6d337b63e4e4bafad3d314b697257: Deleting consensus metadata
I20240506 15:40:03.191794 22978 catalog_manager.cc:4830] TS 7fbbbee6b6744800902a2ae0ef09c15d (127.17.240.195:39087): tablet 2368c4b244fe4807ae92b1ea153d5b1b (table test-table [id=0ae4fd7ec0164668b56915144b06c105]) successfully deleted
W20240506 15:40:03.194139 22976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:46033 (error 108)
I20240506 15:40:03.194661 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.194517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:46033 (local address 127.0.0.1:34200)
0506 15:40:03.194574 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:03.194618 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:46033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20240506 15:40:03.194820 22976 catalog_manager.cc:4573] TS 9a2d68c5893f4e46b48c7802b8816cf1 (127.17.240.193:46033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2368c4b244fe4807ae92b1ea153d5b1b: Network error: Client connection negotiation failed: client connection to 127.17.240.193:46033: connect: Connection refused (error 111)
I20240506 15:40:03.195705 22977 catalog_manager.cc:4830] TS 17f6d337b63e4e4bafad3d314b697257 (127.17.240.194:38545): tablet 2368c4b244fe4807ae92b1ea153d5b1b (table test-table [id=0ae4fd7ec0164668b56915144b06c105]) successfully deleted
I20240506 15:40:03.195854 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 23182
I20240506 15:40:03.200985 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 23313
I20240506 15:40:03.206106 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 22960
2024-05-06T15:40:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (652 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20240506 15:40:03.225977 18371 mini_oidc.cc:152] Starting JWKS server
I20240506 15:40:03.226975 18371 webserver.cc:460] Webserver started at https://127.0.0.1:39881/ using document root <none> and password file <none>
I20240506 15:40:03.227137 18371 mini_oidc.cc:189] Starting OIDC Discovery server
I20240506 15:40:03.227277 18371 webserver.cc:460] Webserver started at http://127.0.0.1:33903/ using document root <none> and password file <none>
2024-05-06T15:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:03Z Disabled control of system clock
I20240506 15:40:03.234155 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36153
--webserver_interface=127.17.240.254
--webserver_port=0
--builtin_ntp_servers=127.17.240.212:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:36153
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39881/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/testchainca.cert with env {}
W20240506 15:40:03.314311 23486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:03.314515 23486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:03.314538 23486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:03.315856 23486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:03.315901 23486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:03.315913 23486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:03.315925 23486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:03.315936 23486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240506 15:40:03.315948 23486 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240506 15:40:03.315958 23486 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39881/jwks.json
I20240506 15:40:03.317626 23486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:36153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39881/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:03.317821 23486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:03.317960 23486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:03.322824 23494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:03.323578 23491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:03.324316 23492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:03.325153 23486 server_base.cc:1034] running on GCE node
I20240506 15:40:03.325353 23486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:03.325569 23486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:03.326735 23486 hybrid_clock.cc:648] HybridClock initialized: now 1715010003326713 us; error 35 us; skew 500 ppm
I20240506 15:40:03.328163 23486 webserver.cc:460] Webserver started at http://127.17.240.254:43585/ using document root <none> and password file <none>
I20240506 15:40:03.328363 23486 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:03.328404 23486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:03.328523 23486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:03.329573 23486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926"
format_stamp: "Formatted at 2024-05-06 15:40:03 on dist-test-slave-v0k1"
server_key: "04d4a06583782039ed3d0a1b0a5f965a"
server_key_iv: "d95d7546b1ed4a40919ab3b231a6e72b"
server_key_version: "encryptionkey@0"
I20240506 15:40:03.329888 23486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926"
format_stamp: "Formatted at 2024-05-06 15:40:03 on dist-test-slave-v0k1"
server_key: "04d4a06583782039ed3d0a1b0a5f965a"
server_key_iv: "d95d7546b1ed4a40919ab3b231a6e72b"
server_key_version: "encryptionkey@0"
I20240506 15:40:03.331146 23486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:40:03.331887 23500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:03.332052 23486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:03.332144 23486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926"
format_stamp: "Formatted at 2024-05-06 15:40:03 on dist-test-slave-v0k1"
server_key: "04d4a06583782039ed3d0a1b0a5f965a"
server_key_iv: "d95d7546b1ed4a40919ab3b231a6e72b"
server_key_version: "encryptionkey@0"
I20240506 15:40:03.332211 23486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:03.345297 23486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:03.345607 23486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:03.345721 23486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:03.349751 23486 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:36153
I20240506 15:40:03.349782 23552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:36153 every 8 connection(s)
I20240506 15:40:03.350097 23486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:40:03.350692 23553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:03.353252 23553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926: Bootstrap starting.
I20240506 15:40:03.353813 23553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:03.354063 23553 log.cc:824] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:03.354729 23553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926: No bootstrap required, opened a new log
I20240506 15:40:03.356191 23553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36153 } }
I20240506 15:40:03.356316 23553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:03.356333 23553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbda8d2b7404c45a6b2d9c1f46e3926, State: Initialized, Role: FOLLOWER
I20240506 15:40:03.356453 23553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36153 } }
I20240506 15:40:03.356531 23553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:40:03.356564 23553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:40:03.356606 23553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:03.357285 23553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36153 } }
I20240506 15:40:03.357393 23553 leader_election.cc:302] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fbda8d2b7404c45a6b2d9c1f46e3926; no voters:
I20240506 15:40:03.357554 23553 leader_election.cc:288] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:40:03.357611 23556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:03.357765 23556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [term 1 LEADER]: Becoming Leader. State: Replica: 2fbda8d2b7404c45a6b2d9c1f46e3926, State: Running, Role: LEADER
I20240506 15:40:03.357792 23553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:03.357867 23556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36153 } }
I20240506 15:40:03.358280 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 23486
I20240506 15:40:03.358407 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:40:03.358597 18371 external_mini_cluster.cc:1428] Setting key 2efe8a4fa9520a13c71720312075bc70
I20240506 15:40:03.358951 23558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fbda8d2b7404c45a6b2d9c1f46e3926. Latest consensus state: current_term: 1 leader_uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36153 } } }
I20240506 15:40:03.359056 23558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:03.359256 23557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbda8d2b7404c45a6b2d9c1f46e3926" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36153 } } }
I20240506 15:40:03.359323 23557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:03.359488 23570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:40:03.359922 23570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:40:03.361289 23570 catalog_manager.cc:1347] Generated new cluster ID: 4fd13bbb9d084c7394ecff4979c1c4b3
I20240506 15:40:03.361351 23570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:40:03.363196 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.359078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.17.240.254:36153)
0506 15:40:03.359460 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.359468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.359492 (+ 24us) server_negotiation.cc:407] Connection header received
0506 15:40:03.359628 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.359634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.359695 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.359846 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:03.360183 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:03.361371 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:03.361988 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:03.362213 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:03.362593 (+ 380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:03.362598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:03.362599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:03.362774 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:03.362802 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:03.362811 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:03.362878 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:03.362951 (+ 73us) server_negotiation.cc:299] Negotiation successful
0506 15:40:03.363000 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"spinlock_wait_cycles":384,"thread_start_us":80,"threads_started":1}
I20240506 15:40:03.364060 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:40:03.377352 23570 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:40:03.377918 23570 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:40:03.382062 23570 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2fbda8d2b7404c45a6b2d9c1f46e3926: Generated new TSK 0
I20240506 15:40:03.382247 23570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:40:03.418586 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.392996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.17.240.254:36153)
0506 15:40:03.393068 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.393070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.393187 (+ 117us) server_negotiation.cc:407] Connection header received
0506 15:40:03.393316 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.393317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.393328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.393390 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0506 15:40:03.393693 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:03.394219 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:03.394840 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:03.395033 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:03.395098 (+ 65us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0506 15:40:03.418278 (+ 23180us) server_negotiation.cc:377] Sending JWT_EXCHANGE NegotiatePB response
0506 15:40:03.418356 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:03.418457 (+ 101us) server_negotiation.cc:299] Negotiation successful
0506 15:40:03.418489 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30,"thread_start_us":123,"threads_started":1}
I20240506 15:40:03.426195 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:03.423443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.17.240.254:36153)
0506 15:40:03.423493 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:03.423495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:03.423626 (+ 131us) server_negotiation.cc:407] Connection header received
0506 15:40:03.423663 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:03.423664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:03.423675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:03.423739 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0506 15:40:03.423969 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:03.424765 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:03.425317 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:03.425520 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:03.425542 (+ 22us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0506 15:40:03.425985 (+ 443us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0506 15:40:03.426125 (+ 140us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59506: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":24}
W20240506 15:40:03.426270 23569 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59506: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20240506 15:40:03.426349 23589 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:03.423364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36153 (local address 127.0.0.1:59506)
0506 15:40:03.423544 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:03.423554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:03.423620 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:03.423742 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:03.423743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:03.423745 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0506 15:40:03.423910 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:03.423912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:03.424848 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:03.424849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:03.425250 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:03.425252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:03.425307 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:03.425312 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0506 15:40:03.426091 (+ 779us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0506 15:40:03.426170 (+ 79us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.240.254:36153: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20240506 15:40:18.433085 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:18.430868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.17.240.254:36153)
0506 15:40:18.430986 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:18.430988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:18.431027 (+ 39us) server_negotiation.cc:407] Connection header received
0506 15:40:18.431132 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:18.431134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:18.431144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:18.431210 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0506 15:40:18.431443 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.431958 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:18.432599 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.432738 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:18.432756 (+ 18us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0506 15:40:18.432927 (+ 171us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0506 15:40:18.433031 (+ 104us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45710: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
W20240506 15:40:18.433161 23595 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45710: JWT verification failed: token expired
W20240506 15:40:18.433167 23594 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:18.430747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36153 (local address 127.0.0.1:45710)
0506 15:40:18.430943 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:18.430954 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:18.431042 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:18.431224 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:18.431225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:18.431227 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0506 15:40:18.431392 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:18.431393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.432042 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:18.432043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:18.432469 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:18.432471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.432530 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:18.432535 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0506 15:40:18.433005 (+ 470us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0506 15:40:18.433090 (+ 85us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.240.254:36153: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
W20240506 15:40:18.434599 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:18.434142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.17.240.254:36153)
0506 15:40:18.434201 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:18.434203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:18.434263 (+ 60us) server_negotiation.cc:407] Connection header received
0506 15:40:18.434350 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:18.434351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:18.434361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:18.434414 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:18.434541 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45718: BlockingRecv error: recv got EOF from 127.0.0.1:45718 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20240506 15:40:18.437340 23605 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.17.240.254:36153
W20240506 15:40:18.437669 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:18.437207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.17.240.254:36153)
0506 15:40:18.437270 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:18.437272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:18.437322 (+ 50us) server_negotiation.cc:407] Connection header received
0506 15:40:18.437457 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:18.437461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:18.437473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:18.437538 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:18.437630 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45730: BlockingRecv error: recv got EOF from 127.0.0.1:45730 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20240506 15:40:18.437786 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 23486
2024-05-06T15:40:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15250 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20240506 15:40:18.476785 18371 mini_oidc.cc:152] Starting JWKS server
I20240506 15:40:18.477537 18371 webserver.cc:460] Webserver started at https://127.0.0.1:43401/ using document root <none> and password file <none>
I20240506 15:40:18.477583 18371 mini_oidc.cc:189] Starting OIDC Discovery server
I20240506 15:40:18.477720 18371 webserver.cc:460] Webserver started at http://127.0.0.1:35219/ using document root <none> and password file <none>
2024-05-06T15:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:18Z Disabled control of system clock
I20240506 15:40:18.485059 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39831
--webserver_interface=127.17.240.254
--webserver_port=0
--builtin_ntp_servers=127.17.240.212:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:39831
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43401/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/testchainca.cert with env {}
W20240506 15:40:18.567934 23611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:18.568142 23611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:18.568164 23611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:18.569520 23611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:18.569573 23611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:18.569586 23611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:18.569598 23611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:18.569609 23611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240506 15:40:18.569622 23611 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240506 15:40:18.569633 23611 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43401/jwks.json
I20240506 15:40:18.571367 23611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:39831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43401/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:18.571579 23611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:18.571725 23611 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:18.577051 23619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:18.577885 23611 server_base.cc:1034] running on GCE node
W20240506 15:40:18.578682 23616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:18.579394 23617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:18.579653 23611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:18.579916 23611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:18.581061 23611 hybrid_clock.cc:648] HybridClock initialized: now 1715010018581049 us; error 34 us; skew 500 ppm
I20240506 15:40:18.588531 23611 webserver.cc:460] Webserver started at http://127.17.240.254:44675/ using document root <none> and password file <none>
I20240506 15:40:18.588769 23611 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:18.588821 23611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:18.588956 23611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:18.590049 23611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "7d5757cec13b42138f1a557166c148e2"
format_stamp: "Formatted at 2024-05-06 15:40:18 on dist-test-slave-v0k1"
server_key: "408306716a08dc4b6dc04f40f9329250"
server_key_iv: "eac2e787592c76259f79dde13b38f4cc"
server_key_version: "encryptionkey@0"
I20240506 15:40:18.590368 23611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "7d5757cec13b42138f1a557166c148e2"
format_stamp: "Formatted at 2024-05-06 15:40:18 on dist-test-slave-v0k1"
server_key: "408306716a08dc4b6dc04f40f9329250"
server_key_iv: "eac2e787592c76259f79dde13b38f4cc"
server_key_version: "encryptionkey@0"
I20240506 15:40:18.591740 23611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:40:18.592548 23625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:18.592763 23611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:18.592847 23611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "7d5757cec13b42138f1a557166c148e2"
format_stamp: "Formatted at 2024-05-06 15:40:18 on dist-test-slave-v0k1"
server_key: "408306716a08dc4b6dc04f40f9329250"
server_key_iv: "eac2e787592c76259f79dde13b38f4cc"
server_key_version: "encryptionkey@0"
I20240506 15:40:18.592913 23611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:18.606493 23611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:18.606809 23611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:18.607081 23611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:18.611353 23611 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:39831
I20240506 15:40:18.611404 23677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:39831 every 8 connection(s)
I20240506 15:40:18.611775 23611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:40:18.612432 23678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:18.615078 23678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2: Bootstrap starting.
I20240506 15:40:18.615662 23678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:18.615900 23678 log.cc:824] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:18.616632 23678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2: No bootstrap required, opened a new log
I20240506 15:40:18.618196 23678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5757cec13b42138f1a557166c148e2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39831 } }
I20240506 15:40:18.618342 23678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:18.618359 23678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5757cec13b42138f1a557166c148e2, State: Initialized, Role: FOLLOWER
I20240506 15:40:18.618466 23678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5757cec13b42138f1a557166c148e2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39831 } }
I20240506 15:40:18.618522 23678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:40:18.618554 23678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:40:18.618605 23678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:18.619334 23678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5757cec13b42138f1a557166c148e2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39831 } }
I20240506 15:40:18.619451 23678 leader_election.cc:302] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d5757cec13b42138f1a557166c148e2; no voters:
I20240506 15:40:18.619601 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 23611
I20240506 15:40:18.619644 23678 leader_election.cc:288] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:40:18.619714 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:40:18.619731 23681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:18.619904 18371 external_mini_cluster.cc:1428] Setting key 6aa92c5b4022f66147ea656ad318b87a
I20240506 15:40:18.619899 23678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:18.619904 23681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [term 1 LEADER]: Becoming Leader. State: Replica: 7d5757cec13b42138f1a557166c148e2, State: Running, Role: LEADER
I20240506 15:40:18.620070 23681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5757cec13b42138f1a557166c148e2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39831 } }
I20240506 15:40:18.620468 23681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d5757cec13b42138f1a557166c148e2. Latest consensus state: current_term: 1 leader_uuid: "7d5757cec13b42138f1a557166c148e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5757cec13b42138f1a557166c148e2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39831 } } }
I20240506 15:40:18.620553 23681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:18.620539 23682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d5757cec13b42138f1a557166c148e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5757cec13b42138f1a557166c148e2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39831 } } }
I20240506 15:40:18.620712 23682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:18.621081 23687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:40:18.621623 23687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:40:18.624665 23687 catalog_manager.cc:1347] Generated new cluster ID: a954194f7758448dbefe70ac7afb5fd1
I20240506 15:40:18.624749 23687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:40:18.626478 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:18.620395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.17.240.254:39831)
0506 15:40:18.622085 (+ 1690us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:18.622093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:18.622120 (+ 27us) server_negotiation.cc:407] Connection header received
0506 15:40:18.622343 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:18.622352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:18.622422 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:18.622593 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:18.623046 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.624175 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:18.625037 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.625711 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:18.625834 (+ 123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:18.625860 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:18.625861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:18.626063 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:18.626091 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:18.626099 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:18.626170 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:18.626244 (+ 74us) server_negotiation.cc:299] Negotiation successful
0506 15:40:18.626298 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":81,"threads_started":1}
I20240506 15:40:18.627543 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:40:18.636850 23687 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:40:18.637632 23687 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:40:18.647226 23687 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7d5757cec13b42138f1a557166c148e2: Generated new TSK 0
I20240506 15:40:18.647460 23687 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240506 15:40:18.674198 23708 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240506 15:40:18.674281 23708 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20240506 15:40:18.674688 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:18.658351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.17.240.254:39831)
0506 15:40:18.658413 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:18.658415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:18.658487 (+ 72us) server_negotiation.cc:407] Connection header received
0506 15:40:18.658588 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:18.658589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:18.658604 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:18.658688 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0506 15:40:18.658962 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.659829 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:18.660487 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.660686 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:18.660754 (+ 68us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0506 15:40:18.674427 (+ 13673us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0506 15:40:18.674597 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":36}
W20240506 15:40:18.674738 23707 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:18.658239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39831 (local address 127.0.0.1:32982)
0506 15:40:18.658419 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:18.658429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:18.658488 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:18.658724 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:18.658724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:18.658727 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0506 15:40:18.658894 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:18.658896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.659965 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:18.659966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:18.660404 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:18.660405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.660473 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:18.660478 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0506 15:40:18.674550 (+ 14072us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0506 15:40:18.674666 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.240.254:39831: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
W20240506 15:40:18.674767 23686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20240506 15:40:18.675321 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 23611
2024-05-06T15:40:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (236 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20240506 15:40:18.714066 18371 mini_oidc.cc:152] Starting JWKS server
I20240506 15:40:18.714866 18371 webserver.cc:460] Webserver started at https://127.0.0.1:40077/ using document root <none> and password file <none>
I20240506 15:40:18.714920 18371 mini_oidc.cc:189] Starting OIDC Discovery server
I20240506 15:40:18.715106 18371 webserver.cc:460] Webserver started at http://127.0.0.1:33349/ using document root <none> and password file <none>
2024-05-06T15:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:18Z Disabled control of system clock
I20240506 15:40:18.722214 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43437
--webserver_interface=127.17.240.254
--webserver_port=0
--builtin_ntp_servers=127.17.240.212:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:43437
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40077/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240506 15:40:18.803426 23714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:18.803651 23714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:18.803682 23714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:18.805105 23714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:18.805158 23714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:18.805171 23714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:18.805184 23714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:18.805195 23714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240506 15:40:18.805208 23714 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240506 15:40:18.805219 23714 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40077/jwks.json
I20240506 15:40:18.807075 23714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:43437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40077/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:18.807327 23714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:18.807562 23714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:18.813330 23722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:18.814069 23720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:18.814880 23719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:18.815773 23714 server_base.cc:1034] running on GCE node
I20240506 15:40:18.816057 23714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:18.816347 23714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:18.817512 23714 hybrid_clock.cc:648] HybridClock initialized: now 1715010018817482 us; error 58 us; skew 500 ppm
I20240506 15:40:18.819216 23714 webserver.cc:460] Webserver started at http://127.17.240.254:44235/ using document root <none> and password file <none>
I20240506 15:40:18.819453 23714 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:18.819504 23714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:18.819633 23714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:18.820698 23714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857"
format_stamp: "Formatted at 2024-05-06 15:40:18 on dist-test-slave-v0k1"
server_key: "f849b82413945ecbcfe47b40d359f927"
server_key_iv: "3a5354ff598b7cf3b02d5dc7541d3bc3"
server_key_version: "encryptionkey@0"
I20240506 15:40:18.821022 23714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857"
format_stamp: "Formatted at 2024-05-06 15:40:18 on dist-test-slave-v0k1"
server_key: "f849b82413945ecbcfe47b40d359f927"
server_key_iv: "3a5354ff598b7cf3b02d5dc7541d3bc3"
server_key_version: "encryptionkey@0"
I20240506 15:40:18.822374 23714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:40:18.823307 23728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:18.823561 23714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:18.823660 23714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857"
format_stamp: "Formatted at 2024-05-06 15:40:18 on dist-test-slave-v0k1"
server_key: "f849b82413945ecbcfe47b40d359f927"
server_key_iv: "3a5354ff598b7cf3b02d5dc7541d3bc3"
server_key_version: "encryptionkey@0"
I20240506 15:40:18.823735 23714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:18.843358 23714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:18.843717 23714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:18.843843 23714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:18.847905 23714 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:43437
I20240506 15:40:18.847976 23780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:43437 every 8 connection(s)
I20240506 15:40:18.848291 23714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:40:18.848918 23781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:18.851572 23781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857: Bootstrap starting.
I20240506 15:40:18.852110 23781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:18.852339 23781 log.cc:824] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:18.853150 23781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857: No bootstrap required, opened a new log
I20240506 15:40:18.854923 23781 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43437 } }
I20240506 15:40:18.855115 23781 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:18.855136 23781 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac67e6d4390d4df5ac5fa7e3ee1de857, State: Initialized, Role: FOLLOWER
I20240506 15:40:18.855255 23781 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43437 } }
I20240506 15:40:18.855330 23781 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:40:18.855356 23781 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:40:18.855383 23781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:18.855996 23781 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43437 } }
I20240506 15:40:18.856097 23781 leader_election.cc:302] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac67e6d4390d4df5ac5fa7e3ee1de857; no voters:
I20240506 15:40:18.856232 23781 leader_election.cc:288] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:40:18.856326 23784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:18.856462 23781 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:18.856495 23784 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [term 1 LEADER]: Becoming Leader. State: Replica: ac67e6d4390d4df5ac5fa7e3ee1de857, State: Running, Role: LEADER
I20240506 15:40:18.856576 23784 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43437 } }
I20240506 15:40:18.856830 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 23714
I20240506 15:40:18.856923 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:40:18.856913 23786 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac67e6d4390d4df5ac5fa7e3ee1de857. Latest consensus state: current_term: 1 leader_uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43437 } } }
I20240506 15:40:18.857007 23786 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:18.857144 18371 external_mini_cluster.cc:1428] Setting key d263920e39be74e1e5ce516af973d30d
I20240506 15:40:18.857254 23791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:40:18.856902 23785 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac67e6d4390d4df5ac5fa7e3ee1de857" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43437 } } }
I20240506 15:40:18.857409 23785 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:18.857766 23791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:40:18.860476 23791 catalog_manager.cc:1347] Generated new cluster ID: 30852bdf94a241c9a593745be84b3bb5
I20240506 15:40:18.860554 23791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:40:18.862588 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:18.857954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.17.240.254:43437)
0506 15:40:18.858266 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:18.858274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:18.858459 (+ 185us) server_negotiation.cc:407] Connection header received
0506 15:40:18.858587 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:18.858593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:18.858657 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:18.858815 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:18.859351 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.860310 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:18.861115 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.861802 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:18.861928 (+ 126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:18.861950 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:18.861950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:18.862170 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:18.862200 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:18.862209 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:18.862292 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:18.862339 (+ 47us) server_negotiation.cc:299] Negotiation successful
0506 15:40:18.862400 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20240506 15:40:18.863610 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:40:18.872498 23791 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:40:18.873059 23791 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:40:18.877938 23791 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ac67e6d4390d4df5ac5fa7e3ee1de857: Generated new TSK 0
I20240506 15:40:18.878222 23791 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240506 15:40:18.902768 23811 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240506 15:40:18.902854 23811 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20240506 15:40:18.904232 23810 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:18.884716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43437 (local address 127.0.0.1:42972)
0506 15:40:18.884912 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:18.884923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:18.884997 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:18.885234 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:18.885236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:18.885239 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0506 15:40:18.885415 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:18.885417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.886776 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:18.886778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:18.887306 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:18.887309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.887382 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:18.887388 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0506 15:40:18.904032 (+ 16644us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40077/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0506 15:40:18.904154 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.240.254:43437: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40077/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":101,"threads_started":1}
I20240506 15:40:18.904254 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:18.884856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.17.240.254:43437)
0506 15:40:18.884942 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:18.884945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:18.884982 (+ 37us) server_negotiation.cc:407] Connection header received
0506 15:40:18.885094 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:18.885095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:18.885110 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:18.885205 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0506 15:40:18.885498 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.886620 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:18.887396 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:18.887614 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:18.887684 (+ 70us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0506 15:40:18.903916 (+ 16232us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40077/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0506 15:40:18.904084 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40077/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20240506 15:40:18.904307 23803 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40077/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20240506 15:40:18.904860 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 23714
2024-05-06T15:40:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2024-05-06T15:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:18Z Disabled control of system clock
I20240506 15:40:18.950146 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39943
--webserver_interface=127.17.240.254
--webserver_port=0
--builtin_ntp_servers=127.17.240.212:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:39943
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/testchainca.cert with env {}
W20240506 15:40:19.049679 23815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:19.049893 23815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:19.049918 23815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:19.051345 23815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:19.051405 23815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:19.051419 23815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:19.051432 23815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:19.051443 23815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240506 15:40:19.051455 23815 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240506 15:40:19.051466 23815 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20240506 15:40:19.053289 23815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:39943
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:19.053529 23815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:19.053699 23815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:19.059162 23820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:19.059811 23823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:19.060549 23821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:19.061362 23815 server_base.cc:1034] running on GCE node
I20240506 15:40:19.061630 23815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:19.061877 23815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:19.063055 23815 hybrid_clock.cc:648] HybridClock initialized: now 1715010019063031 us; error 36 us; skew 500 ppm
I20240506 15:40:19.064584 23815 webserver.cc:460] Webserver started at http://127.17.240.254:33665/ using document root <none> and password file <none>
I20240506 15:40:19.064822 23815 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:19.064875 23815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:19.064991 23815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:19.066053 23815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "f88b5d90600242cfaa264a7c90d9cb47"
format_stamp: "Formatted at 2024-05-06 15:40:19 on dist-test-slave-v0k1"
server_key: "711654d3f546c569a7fee63be0b3365d"
server_key_iv: "74df655d1f16f47f8b8b505ac08dac10"
server_key_version: "encryptionkey@0"
I20240506 15:40:19.066386 23815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "f88b5d90600242cfaa264a7c90d9cb47"
format_stamp: "Formatted at 2024-05-06 15:40:19 on dist-test-slave-v0k1"
server_key: "711654d3f546c569a7fee63be0b3365d"
server_key_iv: "74df655d1f16f47f8b8b505ac08dac10"
server_key_version: "encryptionkey@0"
I20240506 15:40:19.067966 23815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:40:19.068872 23829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:19.069211 23815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:19.069317 23815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "f88b5d90600242cfaa264a7c90d9cb47"
format_stamp: "Formatted at 2024-05-06 15:40:19 on dist-test-slave-v0k1"
server_key: "711654d3f546c569a7fee63be0b3365d"
server_key_iv: "74df655d1f16f47f8b8b505ac08dac10"
server_key_version: "encryptionkey@0"
I20240506 15:40:19.069401 23815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:19.092912 23815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:19.093241 23815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:19.093365 23815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:19.097812 23815 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:39943
I20240506 15:40:19.097859 23881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:39943 every 8 connection(s)
I20240506 15:40:19.098202 23815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:40:19.098961 23882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:19.101547 23882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47: Bootstrap starting.
I20240506 15:40:19.102118 23882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:19.102355 23882 log.cc:824] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:19.103183 23882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47: No bootstrap required, opened a new log
I20240506 15:40:19.104707 23882 raft_consensus.cc:348] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88b5d90600242cfaa264a7c90d9cb47" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39943 } }
I20240506 15:40:19.104861 23882 raft_consensus.cc:374] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:19.104878 23882 raft_consensus.cc:729] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88b5d90600242cfaa264a7c90d9cb47, State: Initialized, Role: FOLLOWER
I20240506 15:40:19.105007 23882 consensus_queue.cc:260] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88b5d90600242cfaa264a7c90d9cb47" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39943 } }
I20240506 15:40:19.105086 23882 raft_consensus.cc:388] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:40:19.105118 23882 raft_consensus.cc:482] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:40:19.105146 23882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:19.105777 23882 raft_consensus.cc:504] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88b5d90600242cfaa264a7c90d9cb47" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39943 } }
I20240506 15:40:19.105883 23882 leader_election.cc:302] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f88b5d90600242cfaa264a7c90d9cb47; no voters:
I20240506 15:40:19.106026 23882 leader_election.cc:288] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:40:19.106150 23885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:19.106274 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 23815
I20240506 15:40:19.106344 23885 raft_consensus.cc:686] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [term 1 LEADER]: Becoming Leader. State: Replica: f88b5d90600242cfaa264a7c90d9cb47, State: Running, Role: LEADER
I20240506 15:40:19.106365 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:40:19.106449 23885 consensus_queue.cc:237] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88b5d90600242cfaa264a7c90d9cb47" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39943 } }
I20240506 15:40:19.106554 18371 external_mini_cluster.cc:1428] Setting key 5b3c7ef9df6cef438dd4cc11ca991c77
I20240506 15:40:19.106783 23885 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f88b5d90600242cfaa264a7c90d9cb47. Latest consensus state: current_term: 1 leader_uuid: "f88b5d90600242cfaa264a7c90d9cb47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88b5d90600242cfaa264a7c90d9cb47" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39943 } } }
I20240506 15:40:19.106794 23886 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f88b5d90600242cfaa264a7c90d9cb47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88b5d90600242cfaa264a7c90d9cb47" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39943 } } }
I20240506 15:40:19.106864 23882 sys_catalog.cc:564] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:19.106909 23886 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:19.106899 23885 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:19.107702 23890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:40:19.108217 23890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:40:19.109867 23890 catalog_manager.cc:1347] Generated new cluster ID: 91254506831a4b88b945b9cece6fc928
I20240506 15:40:19.109925 23890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:40:19.111915 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:19.107183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.17.240.254:39943)
0506 15:40:19.107419 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:19.107427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:19.107470 (+ 43us) server_negotiation.cc:407] Connection header received
0506 15:40:19.107605 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:19.107613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:19.107697 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:19.107855 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:19.108474 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:19.109542 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:19.110363 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:19.110618 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:19.111125 (+ 507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:19.111150 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:19.111151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:40:19.111358 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:19.111391 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:40:19.111399 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:19.111472 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:19.111581 (+ 109us) server_negotiation.cc:299] Negotiation successful
0506 15:40:19.111734 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20240506 15:40:19.112799 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:40:19.122071 23890 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:40:19.123077 23890 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:40:19.149492 23890 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f88b5d90600242cfaa264a7c90d9cb47: Generated new TSK 0
I20240506 15:40:19.149782 23890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:40:19.190071 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:19.158334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.17.240.254:39943)
0506 15:40:19.158388 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:19.158390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:19.158492 (+ 102us) server_negotiation.cc:407] Connection header received
0506 15:40:19.158598 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:19.158599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:19.158614 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:19.158701 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0506 15:40:19.158990 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:19.159834 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:19.160637 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:19.160851 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:19.160924 (+ 73us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0506 15:40:19.189718 (+ 28794us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0506 15:40:19.189956 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45314: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20240506 15:40:19.190164 23891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45314: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20240506 15:40:19.190145 23911 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:19.158218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39943 (local address 127.0.0.1:45314)
0506 15:40:19.158411 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:19.158422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:19.158491 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:19.158730 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:19.158731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:19.158733 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0506 15:40:19.158913 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:19.158915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:19.160082 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:19.160083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:19.160521 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:19.160523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:19.160598 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:19.160605 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0506 15:40:19.189913 (+ 29308us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0506 15:40:19.190063 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.240.254:39943: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20240506 15:40:19.190805 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 23815
2024-05-06T15:40:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (266 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (93 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (94 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:40:19 dist-test-slave-v0k1 krb5kdc[23923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:40:19 dist-test-slave-v0k1 krb5kdc[23923](info): set up 2 sockets
May 06 15:40:19 dist-test-slave-v0k1 krb5kdc[23923](info): commencing operation
krb5kdc: starting...
W20240506 15:40:20.368441 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:40:20 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:40:20.495568 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:37925
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:37925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:20.576326 23936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:20.576545 23936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:20.576581 23936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:20.577926 23936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:20.577981 23936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:20.578004 23936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:20.578022 23936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:20.578038 23936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:40:20.579783 23936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:37925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:37925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:20.580054 23936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:20.580246 23936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:20.585356 23941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:20.586102 23944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:20.587028 23942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:20.587816 23936 server_base.cc:1034] running on GCE node
I20240506 15:40:20.588057 23936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:20.588301 23936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:20.589445 23936 hybrid_clock.cc:648] HybridClock initialized: now 1715010020589435 us; error 30 us; skew 500 ppm
May 06 15:40:20 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010020, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:20.592162 23936 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:40:20.592607 23936 webserver.cc:460] Webserver started at http://127.17.240.254:33451/ using document root <none> and password file <none>
I20240506 15:40:20.592823 23936 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:20.592870 23936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:20.592988 23936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:20.594018 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "e8e44622edda4b5f85212f34789d19e5"
format_stamp: "Formatted at 2024-05-06 15:40:20 on dist-test-slave-v0k1"
server_key: "bbb8c5d785fd62e575420652de621015"
server_key_iv: "f70a8f837ae6e6d3213dc8e67dfbbcfe"
server_key_version: "encryptionkey@0"
I20240506 15:40:20.594332 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "e8e44622edda4b5f85212f34789d19e5"
format_stamp: "Formatted at 2024-05-06 15:40:20 on dist-test-slave-v0k1"
server_key: "bbb8c5d785fd62e575420652de621015"
server_key_iv: "f70a8f837ae6e6d3213dc8e67dfbbcfe"
server_key_version: "encryptionkey@0"
I20240506 15:40:20.595674 23936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:40:20.596413 23951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:20.596614 23936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:20.596706 23936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "e8e44622edda4b5f85212f34789d19e5"
format_stamp: "Formatted at 2024-05-06 15:40:20 on dist-test-slave-v0k1"
server_key: "bbb8c5d785fd62e575420652de621015"
server_key_iv: "f70a8f837ae6e6d3213dc8e67dfbbcfe"
server_key_version: "encryptionkey@0"
I20240506 15:40:20.596760 23936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:20.610927 23936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:20.613242 23936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:20.613355 23936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:20.617345 23936 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:37925
I20240506 15:40:20.617386 24003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:37925 every 8 connection(s)
I20240506 15:40:20.617707 23936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:40:20.618326 24004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:20.620359 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 23936
I20240506 15:40:20.620460 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:40:20.620633 18371 external_mini_cluster.cc:1428] Setting key 9192effdafd748cf5f682c78f4483a3f
I20240506 15:40:20.621042 24004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5: Bootstrap starting.
I20240506 15:40:20.621706 24004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:20.621968 24004 log.cc:824] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:20.622725 24004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5: No bootstrap required, opened a new log
I20240506 15:40:20.624610 24004 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e44622edda4b5f85212f34789d19e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37925 } }
I20240506 15:40:20.624787 24004 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:20.624820 24004 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e44622edda4b5f85212f34789d19e5, State: Initialized, Role: FOLLOWER
I20240506 15:40:20.624938 24004 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e44622edda4b5f85212f34789d19e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37925 } }
I20240506 15:40:20.625018 24004 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:40:20.625057 24004 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:40:20.625159 24004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:20.626011 24004 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e44622edda4b5f85212f34789d19e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37925 } }
I20240506 15:40:20.626142 24004 leader_election.cc:302] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8e44622edda4b5f85212f34789d19e5; no voters:
I20240506 15:40:20.626327 24004 leader_election.cc:288] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:40:20.626410 24009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:20.626627 24004 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:20.626674 24009 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [term 1 LEADER]: Becoming Leader. State: Replica: e8e44622edda4b5f85212f34789d19e5, State: Running, Role: LEADER
I20240506 15:40:20.626825 24009 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e44622edda4b5f85212f34789d19e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37925 } }
May 06 15:40:20 dist-test-slave-v0k1 krb5kdc[23923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:20.627262 24010 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8e44622edda4b5f85212f34789d19e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e44622edda4b5f85212f34789d19e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37925 } } }
I20240506 15:40:20.627297 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e44622edda4b5f85212f34789d19e5. Latest consensus state: current_term: 1 leader_uuid: "e8e44622edda4b5f85212f34789d19e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e44622edda4b5f85212f34789d19e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37925 } } }
I20240506 15:40:20.627353 24010 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:20.627383 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:20.627664 24017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:40:20.628533 24017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:40:20.630548 24017 catalog_manager.cc:1347] Generated new cluster ID: a75e1d8101334cda9fd3366aa7afd33c
I20240506 15:40:20.630607 24017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:40:20.631649 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:20.621185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.17.240.254:37925)
0506 15:40:20.621361 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:20.621371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:20.621390 (+ 19us) server_negotiation.cc:407] Connection header received
0506 15:40:20.621857 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:20.621866 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:20.621913 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:20.622053 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:20.622960 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:20.623778 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:20.624682 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:20.624993 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:20.627676 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:20.627687 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:20.627688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:20.627706 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:20.629725 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:20.630180 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:20.630183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:20.630184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:20.630244 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:20.630415 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:20.630416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:20.630416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:20.630712 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:20.630855 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:20.631315 (+ 460us) server_negotiation.cc:299] Negotiation successful
0506 15:40:20.631430 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20240506 15:40:20.661631 24017 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:40:20.662403 24017 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:40:20.666337 24017 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e8e44622edda4b5f85212f34789d19e5: Generated new TSK 0
I20240506 15:40:20.666554 24017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:40:20.698081 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:37925
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:20.779970 24030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:20.780161 24030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:20.780184 24030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:20.781513 24030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:20.781569 24030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:20.781611 24030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:40:20.783351 24030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:37925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:20.783558 24030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:20.783748 24030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:20.789012 24038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:20.789525 24036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:20.790396 24030 server_base.cc:1034] running on GCE node
W20240506 15:40:20.790997 24035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:20.791309 24030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:20.791568 24030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:20.792752 24030 hybrid_clock.cc:648] HybridClock initialized: now 1715010020792732 us; error 34 us; skew 500 ppm
May 06 15:40:20 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010020, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:20.795469 24030 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:40:20.795936 24030 webserver.cc:460] Webserver started at http://127.17.240.193:46447/ using document root <none> and password file <none>
I20240506 15:40:20.796123 24030 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:20.796164 24030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:20.796263 24030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:20.797242 24030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "3717ef83d4004d51921a4b72526e17f1"
format_stamp: "Formatted at 2024-05-06 15:40:20 on dist-test-slave-v0k1"
server_key: "3b17bfdfd08af3119a036932368d1b15"
server_key_iv: "7c25e38caea44530296c881aba71a5e8"
server_key_version: "encryptionkey@0"
I20240506 15:40:20.797552 24030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "3717ef83d4004d51921a4b72526e17f1"
format_stamp: "Formatted at 2024-05-06 15:40:20 on dist-test-slave-v0k1"
server_key: "3b17bfdfd08af3119a036932368d1b15"
server_key_iv: "7c25e38caea44530296c881aba71a5e8"
server_key_version: "encryptionkey@0"
I20240506 15:40:20.798847 24030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:40:20.799690 24045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:20.799890 24030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:20.799962 24030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "3717ef83d4004d51921a4b72526e17f1"
format_stamp: "Formatted at 2024-05-06 15:40:20 on dist-test-slave-v0k1"
server_key: "3b17bfdfd08af3119a036932368d1b15"
server_key_iv: "7c25e38caea44530296c881aba71a5e8"
server_key_version: "encryptionkey@0"
I20240506 15:40:20.800019 24030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:20.811990 24030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:20.814260 24030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:20.814370 24030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:20.814545 24030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:20.814878 24030 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:20.814914 24030 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:20.814937 24030 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:20.814961 24030 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:20.820238 24030 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:41623
I20240506 15:40:20.820302 24158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:41623 every 8 connection(s)
I20240506 15:40:20.820598 24030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:40:20.823145 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 24030
I20240506 15:40:20.823263 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:40:20.823419 18371 external_mini_cluster.cc:1428] Setting key 113d95f5faa0d93bb02943181ca7313f
May 06 15:40:20 dist-test-slave-v0k1 krb5kdc[23923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010020, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:20.830284 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:20.821360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:55201 (local address 127.17.240.254:37925)
0506 15:40:20.821425 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:20.821427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:20.821628 (+ 201us) server_negotiation.cc:407] Connection header received
0506 15:40:20.822066 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:20.822067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:20.822078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:20.822144 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:20.822957 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:20.823729 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:20.824479 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:20.824712 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:20.827085 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:20.827089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:20.827090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:20.827098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:20.828965 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:20.829401 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:20.829403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:20.829403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:20.829438 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:20.829625 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:20.829626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:20.829626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:20.829733 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:20.829791 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:20.830059 (+ 268us) server_negotiation.cc:299] Negotiation successful
0506 15:40:20.830172 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240506 15:40:20.830399 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:20.821331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:37925 (local address 127.17.240.193:55201)
0506 15:40:20.821482 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:20.821513 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:20.821968 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:20.822188 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:20.822191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:20.822516 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:20.822817 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:20.822821 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:20.823853 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:20.823854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:20.824373 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:20.824376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:20.824721 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:20.825292 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:20.825310 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:20.826958 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:20.829064 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:20.829074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:20.829078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:20.829327 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:20.829499 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:20.829502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:20.829502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:20.829571 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:20.829792 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:20.829798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:20.829929 (+ 131us) client_negotiation.cc:769] Sending connection context
0506 15:40:20.830034 (+ 105us) client_negotiation.cc:240] Negotiation successful
0506 15:40:20.830194 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20240506 15:40:20.830930 24159 heartbeater.cc:347] Connected to a master server at 127.17.240.254:37925
I20240506 15:40:20.831081 24159 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:20.831358 24159 heartbeater.cc:511] Master 127.17.240.254:37925 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
I20240506 15:40:20.831941 23968 ts_manager.cc:194] Registered new tserver with Master: 3717ef83d4004d51921a4b72526e17f1 (127.17.240.193:41623)
I20240506 15:40:20.832612 23968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:55201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:40:20.851375 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:37925
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:20.934881 24164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:20.935114 24164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:20.935140 24164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:20.936540 24164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:20.936595 24164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:20.936621 24164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:40:20.938297 24164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:37925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:20.938506 24164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:20.938669 24164 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:20.944036 24169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:20.944584 24172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:20.945336 24170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:20.946149 24164 server_base.cc:1034] running on GCE node
I20240506 15:40:20.946378 24164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:20.946630 24164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:20.947777 24164 hybrid_clock.cc:648] HybridClock initialized: now 1715010020947755 us; error 42 us; skew 500 ppm
May 06 15:40:20 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010020, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:20.950403 24164 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:40:20.950857 24164 webserver.cc:460] Webserver started at http://127.17.240.194:35073/ using document root <none> and password file <none>
I20240506 15:40:20.951072 24164 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:20.951122 24164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:20.951217 24164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:20.952232 24164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "51b810ea0f074597a732e4e00abb4442"
format_stamp: "Formatted at 2024-05-06 15:40:20 on dist-test-slave-v0k1"
server_key: "d5ecaaffd713a6fcf530b0507a870d02"
server_key_iv: "64de4bbc3fbd511a349dcbbdcc6c6bc0"
server_key_version: "encryptionkey@0"
I20240506 15:40:20.952528 24164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "51b810ea0f074597a732e4e00abb4442"
format_stamp: "Formatted at 2024-05-06 15:40:20 on dist-test-slave-v0k1"
server_key: "d5ecaaffd713a6fcf530b0507a870d02"
server_key_iv: "64de4bbc3fbd511a349dcbbdcc6c6bc0"
server_key_version: "encryptionkey@0"
I20240506 15:40:20.953814 24164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:40:20.954464 24179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:20.954676 24164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:20.954757 24164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "51b810ea0f074597a732e4e00abb4442"
format_stamp: "Formatted at 2024-05-06 15:40:20 on dist-test-slave-v0k1"
server_key: "d5ecaaffd713a6fcf530b0507a870d02"
server_key_iv: "64de4bbc3fbd511a349dcbbdcc6c6bc0"
server_key_version: "encryptionkey@0"
I20240506 15:40:20.954823 24164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:21.016103 24164 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:21.018528 24164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:21.018662 24164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:21.018903 24164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:21.019279 24164 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:21.019315 24164 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:21.019351 24164 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:21.019378 24164 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:21.024572 24164 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:34981
I20240506 15:40:21.024628 24292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:34981 every 8 connection(s)
I20240506 15:40:21.024925 24164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:40:21.027485 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 24164
I20240506 15:40:21.027590 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:40:21.027737 18371 external_mini_cluster.cc:1428] Setting key ffc680d5fd398cd6df1a9a7a50ad2728
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010020, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:21.033871 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:21.025577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:41749 (local address 127.17.240.254:37925)
0506 15:40:21.025634 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:21.025637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:21.025806 (+ 169us) server_negotiation.cc:407] Connection header received
0506 15:40:21.026341 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:21.026343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:21.026356 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:21.026417 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:21.027313 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.027898 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:21.028442 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.028642 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:21.030918 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:21.030925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:21.030925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:21.030932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:21.032646 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:21.033052 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:21.033053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:21.033054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:21.033081 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:21.033254 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:21.033255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:21.033255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:21.033358 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:21.033419 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:21.033675 (+ 256us) server_negotiation.cc:299] Negotiation successful
0506 15:40:21.033774 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240506 15:40:21.034024 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:21.025562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:37925 (local address 127.17.240.194:41749)
0506 15:40:21.025698 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:21.025724 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:21.026237 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:21.026479 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:21.026484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:21.026832 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:21.027208 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:21.027212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.027972 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:21.027973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:21.028364 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:21.028366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.028755 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:21.029312 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:21.029328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:21.030814 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:21.032744 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:21.032749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:21.032753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:21.032971 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:21.033138 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:21.033139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:21.033140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:21.033199 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:21.033415 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:21.033420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:21.033548 (+ 128us) client_negotiation.cc:769] Sending connection context
0506 15:40:21.033650 (+ 102us) client_negotiation.cc:240] Negotiation successful
0506 15:40:21.033794 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20240506 15:40:21.034464 24293 heartbeater.cc:347] Connected to a master server at 127.17.240.254:37925
I20240506 15:40:21.034593 24293 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:21.034843 24293 heartbeater.cc:511] Master 127.17.240.254:37925 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
I20240506 15:40:21.035301 23968 ts_manager.cc:194] Registered new tserver with Master: 51b810ea0f074597a732e4e00abb4442 (127.17.240.194:34981)
I20240506 15:40:21.035687 23968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:41749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:40:21.054813 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:37925
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:21.137739 24298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:21.137944 24298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:21.137969 24298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:21.139397 24298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:21.139454 24298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:21.139482 24298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:40:21.141185 24298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:37925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:21.141412 24298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:21.141618 24298 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:21.147114 24304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:21.147771 24303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:21.148496 24306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:21.149310 24298 server_base.cc:1034] running on GCE node
I20240506 15:40:21.149541 24298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:21.149782 24298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:21.150943 24298 hybrid_clock.cc:648] HybridClock initialized: now 1715010021150917 us; error 50 us; skew 500 ppm
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:21.153671 24298 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:40:21.154124 24298 webserver.cc:460] Webserver started at http://127.17.240.195:39593/ using document root <none> and password file <none>
I20240506 15:40:21.154314 24298 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:21.154356 24298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:21.154451 24298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:21.155565 24298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "dbe21ad607584ef2bc0c955fd170e6b2"
format_stamp: "Formatted at 2024-05-06 15:40:21 on dist-test-slave-v0k1"
server_key: "5b46fc819da3850630387f77d58c3d16"
server_key_iv: "f0297290c7451041b58021a781269171"
server_key_version: "encryptionkey@0"
I20240506 15:40:21.155892 24298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "dbe21ad607584ef2bc0c955fd170e6b2"
format_stamp: "Formatted at 2024-05-06 15:40:21 on dist-test-slave-v0k1"
server_key: "5b46fc819da3850630387f77d58c3d16"
server_key_iv: "f0297290c7451041b58021a781269171"
server_key_version: "encryptionkey@0"
I20240506 15:40:21.157181 24298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:40:21.158020 24313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:21.158247 24298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:21.158357 24298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "dbe21ad607584ef2bc0c955fd170e6b2"
format_stamp: "Formatted at 2024-05-06 15:40:21 on dist-test-slave-v0k1"
server_key: "5b46fc819da3850630387f77d58c3d16"
server_key_iv: "f0297290c7451041b58021a781269171"
server_key_version: "encryptionkey@0"
I20240506 15:40:21.158432 24298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:21.173879 24298 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:21.176374 24298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:21.176486 24298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:21.176683 24298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:21.177006 24298 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:21.177047 24298 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:21.177069 24298 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:21.177083 24298 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:21.183827 24298 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:39287
I20240506 15:40:21.183916 24426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:39287 every 8 connection(s)
I20240506 15:40:21.184198 24298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:21.189955 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 24298
I20240506 15:40:21.190048 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:40:21.190218 18371 external_mini_cluster.cc:1428] Setting key 716cd6abb789af2c1a12555dffa6173c
I20240506 15:40:21.192807 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:21.184903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:38937 (local address 127.17.240.254:37925)
0506 15:40:21.184978 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:21.184980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:21.185148 (+ 168us) server_negotiation.cc:407] Connection header received
0506 15:40:21.185654 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:21.185654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:21.185667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:21.185728 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:21.186512 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.187045 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:21.187606 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.187793 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:21.190054 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:21.190059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:21.190060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:21.190068 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:21.191602 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:21.191987 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:21.191988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:21.191989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:21.192015 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:21.192195 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:21.192196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:21.192196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:21.192302 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:21.192361 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:21.192614 (+ 253us) server_negotiation.cc:299] Negotiation successful
0506 15:40:21.192712 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240506 15:40:21.192945 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:21.184828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:37925 (local address 127.17.240.195:38937)
0506 15:40:21.185018 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:21.185065 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:21.185529 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:21.185797 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:21.185801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:21.186127 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:21.186408 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:21.186413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.187133 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:21.187134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:21.187516 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:21.187517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.187947 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:21.188518 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:21.188533 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:21.189953 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:21.191689 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:21.191695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:21.191698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:21.191918 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:21.192076 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:21.192077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:21.192077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:21.192140 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:21.192358 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:21.192363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:21.192485 (+ 122us) client_negotiation.cc:769] Sending connection context
0506 15:40:21.192588 (+ 103us) client_negotiation.cc:240] Negotiation successful
0506 15:40:21.192737 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20240506 15:40:21.193770 24427 heartbeater.cc:347] Connected to a master server at 127.17.240.254:37925
I20240506 15:40:21.193967 24427 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:21.194289 24427 heartbeater.cc:511] Master 127.17.240.254:37925 requested a full tablet report, sending...
I20240506 15:40:21.194778 23968 ts_manager.cc:194] Registered new tserver with Master: dbe21ad607584ef2bc0c955fd170e6b2 (127.17.240.195:39287)
I20240506 15:40:21.195240 23968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:38937
I20240506 15:40:21.202286 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/test-admin.keytab.
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:21.220185 18371 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20240506 15:40:21.221820 24434 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:21.221063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.17.240.193:41623)
0506 15:40:21.221229 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:21.221237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:21.221244 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:40:21.221271 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:21.221272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:21.221285 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:21.221354 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:21.221719 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58318: BlockingRecv error: recv got EOF from 127.0.0.1:58318 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
W20240506 15:40:21.221930 18371 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5cc' of type FILE
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:21.222903 18371 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:21.223834 18371 init.cc:295] Successfully reacquired a new kerberos TGT
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:40:21.231446 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:21.224406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.17.240.193:41623)
0506 15:40:21.224454 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:21.224455 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:21.224501 (+ 46us) server_negotiation.cc:407] Connection header received
0506 15:40:21.224551 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:21.224552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:21.224562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:21.224614 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:21.225125 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.225844 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:21.226534 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.226770 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:21.228787 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:21.228798 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:21.228798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:21.228804 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:21.230466 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:21.230797 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:21.230799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:21.230799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:21.230823 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:21.230988 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:21.230989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:21.230989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:21.231121 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:21.231235 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:21.231304 (+ 69us) server_negotiation.cc:299] Negotiation successful
0506 15:40:21.231367 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20240506 15:40:21.233144 24434 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:21.232654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.17.240.193:41623)
0506 15:40:21.232712 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:21.232714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:21.232759 (+ 45us) server_negotiation.cc:407] Connection header received
0506 15:40:21.232809 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:21.232809 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:21.232818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:21.232862 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:21.233104 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58338: BlockingRecv error: recv got EOF from 127.0.0.1:58338 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20240506 15:40:21.233237 18371 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5cc' of type FILE
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:21.234342 18371 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:21.235306 18371 init.cc:295] Successfully reacquired a new kerberos TGT
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:40:21.241863 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:21.235892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.17.240.193:41623)
0506 15:40:21.235940 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:21.235942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:21.235949 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:40:21.235969 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:21.235969 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:21.235977 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:21.236028 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:21.236472 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.236962 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:21.237476 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.237604 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:21.239586 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:21.239590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:21.239590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:21.239596 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:21.240949 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:21.241262 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:21.241264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:21.241264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:21.241289 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:21.241423 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:21.241424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:21.241424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:21.241499 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:21.241568 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:21.241655 (+ 87us) server_negotiation.cc:299] Negotiation successful
0506 15:40:21.241725 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20240506 15:40:21.243381 24434 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:21.242891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.17.240.193:41623)
0506 15:40:21.242932 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:21.242934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:21.242982 (+ 48us) server_negotiation.cc:407] Connection header received
0506 15:40:21.243045 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:21.243046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:21.243054 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:21.243112 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:21.243334 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58364: BlockingRecv error: recv got EOF from 127.0.0.1:58364 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20240506 15:40:21.243515 18371 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715009978772892-18371-0/krb5kdc/krb5cc' of type FILE
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:21.244451 18371 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:21.245213 18371 init.cc:295] Successfully reacquired a new kerberos TGT
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[23923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:40:21.251652 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:21.245742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.17.240.193:41623)
0506 15:40:21.245790 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:21.245792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:21.245799 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:40:21.245820 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:21.245821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:21.245830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:21.245875 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:21.246343 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.246784 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:21.247365 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:21.247541 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:21.249362 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:21.249365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:21.249366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:21.249372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:21.250583 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:21.250858 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:21.250860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:21.250860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:21.250882 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:21.251030 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:21.251031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:21.251031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:21.251209 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:21.251330 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:21.251454 (+ 124us) server_negotiation.cc:299] Negotiation successful
0506 15:40:21.251560 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:40:21.252229 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 24030
I20240506 15:40:21.256986 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 24164
I20240506 15:40:21.261878 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 24298
I20240506 15:40:21.266470 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 23936
2024-05-06T15:40:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1889 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[24447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[24447](info): set up 2 sockets
May 06 15:40:21 dist-test-slave-v0k1 krb5kdc[24447](info): commencing operation
krb5kdc: starting...
W20240506 15:40:22.223544 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:40:22 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.254.keytab.
Entry for principal oryx/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.254.keytab.
I20240506 15:40:22.342419 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43707
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.254.keytab
--principal=oryx/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:43707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:22.422765 24460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:22.422991 24460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:22.423054 24460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:22.424402 24460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:22.424451 24460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:22.424474 24460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:22.424494 24460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:22.424513 24460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:40:22.426033 24460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:43707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.254.keytab
--principal=oryx/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:22.426276 24460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:22.426465 24460 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:22.431370 24466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:22.432210 24468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:22.432950 24465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:22.433745 24460 server_base.cc:1034] running on GCE node
I20240506 15:40:22.433988 24460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:22.434252 24460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:22.435415 24460 hybrid_clock.cc:648] HybridClock initialized: now 1715010022435408 us; error 36 us; skew 500 ppm
May 06 15:40:22 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:22.437819 24460 init.cc:377] Logged in from keytab as oryx/127.17.240.254@KRBTEST.COM (short username oryx)
I20240506 15:40:22.438258 24460 webserver.cc:460] Webserver started at http://127.17.240.254:42567/ using document root <none> and password file <none>
I20240506 15:40:22.438436 24460 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:22.438480 24460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:22.438593 24460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:22.439631 24460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "0a137b87b00841f5a73fc1d4078fc43c"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "3839dcd5d15250a1fd03f51a13025d40"
server_key_iv: "31f403a0870013de680f9cfaf5b11666"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.439934 24460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "0a137b87b00841f5a73fc1d4078fc43c"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "3839dcd5d15250a1fd03f51a13025d40"
server_key_iv: "31f403a0870013de680f9cfaf5b11666"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.441228 24460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:40:22.441932 24475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:22.442095 24460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:22.442181 24460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "0a137b87b00841f5a73fc1d4078fc43c"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "3839dcd5d15250a1fd03f51a13025d40"
server_key_iv: "31f403a0870013de680f9cfaf5b11666"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.442241 24460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:22.453238 24460 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:22.455248 24460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:22.455377 24460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:22.459168 24460 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:43707
I20240506 15:40:22.459218 24527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:43707 every 8 connection(s)
I20240506 15:40:22.459522 24460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:40:22.460345 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:22.462460 24528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c: Bootstrap starting.
I20240506 15:40:22.462998 24528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:22.463307 24528 log.cc:824] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:22.463989 24528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c: No bootstrap required, opened a new log
I20240506 15:40:22.465410 24528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } }
I20240506 15:40:22.465533 24528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:22.465550 24528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a137b87b00841f5a73fc1d4078fc43c, State: Initialized, Role: FOLLOWER
I20240506 15:40:22.465668 24528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } }
I20240506 15:40:22.465740 24528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:40:22.465776 24528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:40:22.465845 24528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:22.466414 24528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } }
I20240506 15:40:22.466506 24528 leader_election.cc:302] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a137b87b00841f5a73fc1d4078fc43c; no voters:
I20240506 15:40:22.466663 24528 leader_election.cc:288] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:40:22.466874 24531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:22.466928 24528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:22.467037 24531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 1 LEADER]: Becoming Leader. State: Replica: 0a137b87b00841f5a73fc1d4078fc43c, State: Running, Role: LEADER
I20240506 15:40:22.467204 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 24460
I20240506 15:40:22.467192 24531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } }
I20240506 15:40:22.467308 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:40:22.467497 18371 external_mini_cluster.cc:1428] Setting key 1213f6fffb787a8bd729df303928776a
I20240506 15:40:22.467793 24533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a137b87b00841f5a73fc1d4078fc43c. Latest consensus state: current_term: 1 leader_uuid: "0a137b87b00841f5a73fc1d4078fc43c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } }
I20240506 15:40:22.467886 24533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:22.467777 24532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a137b87b00841f5a73fc1d4078fc43c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } }
I20240506 15:40:22.468047 24532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: This master's current role is: LEADER
W20240506 15:40:22.468820 24549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:40:22.468909 24549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:40:22.468968 24541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:40:22.469127 24541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:40:22.470530 24541 catalog_manager.cc:1347] Generated new cluster ID: 2798417bad99443ebbf226345980009d
I20240506 15:40:22.470585 24541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:40:22 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:22.476835 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:22.468141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.17.240.254:43707)
0506 15:40:22.468320 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:22.468334 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:22.468362 (+ 28us) server_negotiation.cc:407] Connection header received
0506 15:40:22.468485 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:22.468492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:22.468605 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:22.468770 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:22.469588 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.470586 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:22.471286 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.471917 (+ 631us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:22.473740 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:22.473746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:22.473746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:22.473762 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:22.475512 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:22.475836 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:22.475838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:22.475839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:22.475886 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:22.476045 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:22.476046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:22.476046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:22.476238 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:22.476364 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:22.476488 (+ 124us) server_negotiation.cc:299] Negotiation successful
0506 15:40:22.476638 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20240506 15:40:22.482123 24541 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:40:22.482851 24541 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:40:22.499222 24541 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c: Generated new TSK 0
I20240506 15:40:22.499439 24541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab.
Entry for principal oryx/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab.
I20240506 15:40:22.530913 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43707
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab
--principal=oryx/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:22.630735 24554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:22.630939 24554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:22.630964 24554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:22.632344 24554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:22.632391 24554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:22.632417 24554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:40:22.633880 24554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab
--principal=oryx/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:22.634083 24554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:22.634282 24554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:22.639544 24559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:22.640197 24562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:22.640899 24560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:22.641742 24554 server_base.cc:1034] running on GCE node
I20240506 15:40:22.641932 24554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:22.642161 24554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:22.643311 24554 hybrid_clock.cc:648] HybridClock initialized: now 1715010022643291 us; error 43 us; skew 500 ppm
May 06 15:40:22 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:22.645937 24554 init.cc:377] Logged in from keytab as oryx/127.17.240.193@KRBTEST.COM (short username oryx)
I20240506 15:40:22.646399 24554 webserver.cc:460] Webserver started at http://127.17.240.193:46801/ using document root <none> and password file <none>
I20240506 15:40:22.646590 24554 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:22.646637 24554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:22.646749 24554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:22.647790 24554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "9899ac0aff6c40b1bdb86a2f5d922d76"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "91e15d8ffd3b430ef6afe54781fb2429"
server_key_iv: "e18ca85b4d4af716cf472f6b2c6ade09"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.648094 24554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "9899ac0aff6c40b1bdb86a2f5d922d76"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "91e15d8ffd3b430ef6afe54781fb2429"
server_key_iv: "e18ca85b4d4af716cf472f6b2c6ade09"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.649286 24554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:40:22.649957 24569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:22.650110 24554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:40:22.650189 24554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "9899ac0aff6c40b1bdb86a2f5d922d76"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "91e15d8ffd3b430ef6afe54781fb2429"
server_key_iv: "e18ca85b4d4af716cf472f6b2c6ade09"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.650256 24554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:22.658480 24554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:22.660300 24554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:22.660403 24554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:22.660629 24554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:22.660959 24554 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:22.660998 24554 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:22.661033 24554 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:22.661065 24554 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:22.666569 24554 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:36253
I20240506 15:40:22.666633 24682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:36253 every 8 connection(s)
I20240506 15:40:22.666966 24554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
May 06 15:40:22 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:22.675346 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:22.667736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:50325 (local address 127.17.240.254:43707)
0506 15:40:22.667810 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:22.667812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:22.668029 (+ 217us) server_negotiation.cc:407] Connection header received
0506 15:40:22.668529 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:22.668530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:22.668547 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:22.668614 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:22.669669 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.670157 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:22.670697 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.670865 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:22.672862 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:22.672865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:22.672866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:22.672873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:22.674118 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:22.674498 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:22.674499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:22.674500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:22.674525 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:22.674704 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:22.674705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:22.674705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:22.674796 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:22.674856 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:22.675137 (+ 281us) server_negotiation.cc:299] Negotiation successful
0506 15:40:22.675244 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240506 15:40:22.675457 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:22.667705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:50325)
0506 15:40:22.667854 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:22.667884 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:22.668421 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:22.668678 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:22.668682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:22.668988 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:22.669553 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:22.669557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.670249 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:22.670250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:22.670616 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:22.670617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.670780 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:22.671365 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:22.671381 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:22.672765 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:22.674222 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:22.674229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:22.674232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:22.674431 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:22.674582 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:22.674582 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:22.674583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:22.674642 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:22.674851 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:22.674856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:22.674973 (+ 117us) client_negotiation.cc:769] Sending connection context
0506 15:40:22.675113 (+ 140us) client_negotiation.cc:240] Negotiation successful
0506 15:40:22.675260 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20240506 15:40:22.676008 24683 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43707
I20240506 15:40:22.676106 24683 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:22.676286 24683 heartbeater.cc:511] Master 127.17.240.254:43707 requested a full tablet report, sending...
I20240506 15:40:22.676488 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 24554
I20240506 15:40:22.676599 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:40:22.676791 18371 external_mini_cluster.cc:1428] Setting key bbcb77a5d7116924dc85cf6dabd10e03
I20240506 15:40:22.676862 24492 ts_manager.cc:194] Registered new tserver with Master: 9899ac0aff6c40b1bdb86a2f5d922d76 (127.17.240.193:36253)
I20240506 15:40:22.677647 24492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:50325
WARNING: no policy specified for oryx/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab.
Entry for principal oryx/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab.
I20240506 15:40:22.702523 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43707
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab
--principal=oryx/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:22.782256 24688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:22.782470 24688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:22.782495 24688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:22.783856 24688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:22.783906 24688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:22.783932 24688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:40:22.785395 24688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab
--principal=oryx/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:22.785575 24688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:22.785728 24688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:22.790884 24693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:22.791744 24694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:22.792452 24696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:22.793299 24688 server_base.cc:1034] running on GCE node
I20240506 15:40:22.793474 24688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:22.793699 24688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:22.794832 24688 hybrid_clock.cc:648] HybridClock initialized: now 1715010022794812 us; error 47 us; skew 500 ppm
May 06 15:40:22 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:22.797425 24688 init.cc:377] Logged in from keytab as oryx/127.17.240.194@KRBTEST.COM (short username oryx)
I20240506 15:40:22.797896 24688 webserver.cc:460] Webserver started at http://127.17.240.194:41703/ using document root <none> and password file <none>
I20240506 15:40:22.798079 24688 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:22.798123 24688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:22.798254 24688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:22.799317 24688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "780c96a5e30c48ecb3fff58900d1a0bf"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "ebdedb47eda70254c1f220008b5ae825"
server_key_iv: "bc135f6e53fb78a4c87498320dc8d8ac"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.799634 24688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "780c96a5e30c48ecb3fff58900d1a0bf"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "ebdedb47eda70254c1f220008b5ae825"
server_key_iv: "bc135f6e53fb78a4c87498320dc8d8ac"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.800863 24688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:40:22.801524 24703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:22.801697 24688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:22.801774 24688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "780c96a5e30c48ecb3fff58900d1a0bf"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "ebdedb47eda70254c1f220008b5ae825"
server_key_iv: "bc135f6e53fb78a4c87498320dc8d8ac"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.801836 24688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:22.824586 24688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:22.826797 24688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:22.826934 24688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:22.827239 24688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:22.827566 24688 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:22.827600 24688 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:22.827636 24688 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:22.827670 24688 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:22.833395 24688 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:38169
I20240506 15:40:22.833438 24816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:38169 every 8 connection(s)
I20240506 15:40:22.833747 24688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:40:22.837399 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 24688
I20240506 15:40:22.837517 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:40:22.837724 18371 external_mini_cluster.cc:1428] Setting key c1f4f16dc78d287eebd80a2aa170c20f
May 06 15:40:22 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:22.842702 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:22.834430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:33361 (local address 127.17.240.254:43707)
0506 15:40:22.834509 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:22.834512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:22.834652 (+ 140us) server_negotiation.cc:407] Connection header received
0506 15:40:22.835203 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:22.835204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:22.835217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:22.835277 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:22.836423 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.836917 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:22.837470 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.837641 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:22.839985 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:22.839989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:22.839989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:22.839996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:22.841455 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:22.841883 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:22.841887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:22.841887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:22.841911 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:22.842089 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:22.842090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:22.842090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:22.842191 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:22.842252 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:22.842508 (+ 256us) server_negotiation.cc:299] Negotiation successful
0506 15:40:22.842610 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240506 15:40:22.842841 24819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:22.834381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:33361)
0506 15:40:22.834543 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:22.834569 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:22.835093 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:22.835341 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:22.835345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:22.835750 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:22.836317 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:22.836322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.837001 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:22.837004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:22.837394 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:22.837395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.837531 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:22.838218 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:22.838234 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:22.839865 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:22.841579 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:22.841586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:22.841589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:22.841808 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:22.841968 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:22.841969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:22.841969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:22.842034 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:22.842247 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:22.842252 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:22.842378 (+ 126us) client_negotiation.cc:769] Sending connection context
0506 15:40:22.842484 (+ 106us) client_negotiation.cc:240] Negotiation successful
0506 15:40:22.842638 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20240506 15:40:22.843329 24817 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43707
I20240506 15:40:22.843426 24817 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:22.843634 24817 heartbeater.cc:511] Master 127.17.240.254:43707 requested a full tablet report, sending...
I20240506 15:40:22.844015 24492 ts_manager.cc:194] Registered new tserver with Master: 780c96a5e30c48ecb3fff58900d1a0bf (127.17.240.194:38169)
I20240506 15:40:22.844406 24492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:33361
WARNING: no policy specified for oryx/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab.
Entry for principal oryx/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab.
I20240506 15:40:22.866297 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43707
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab
--principal=oryx/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:22.945544 24822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:22.945760 24822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:22.945784 24822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:22.947161 24822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:22.947209 24822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:22.947235 24822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:40:22.948706 24822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab
--principal=oryx/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:22.948879 24822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:22.949029 24822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:22.954176 24830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:22.954788 24827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:22.955574 24828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:22.956408 24822 server_base.cc:1034] running on GCE node
I20240506 15:40:22.956635 24822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:22.956857 24822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:22.957994 24822 hybrid_clock.cc:648] HybridClock initialized: now 1715010022957977 us; error 36 us; skew 500 ppm
May 06 15:40:22 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:22.960623 24822 init.cc:377] Logged in from keytab as oryx/127.17.240.195@KRBTEST.COM (short username oryx)
I20240506 15:40:22.961104 24822 webserver.cc:460] Webserver started at http://127.17.240.195:37273/ using document root <none> and password file <none>
I20240506 15:40:22.961289 24822 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:22.961337 24822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:22.961462 24822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:22.962476 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "eed1081d2c3e466393ce04310c6f5579"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "fb976d0540dcaf1f09f8090f98ece94e"
server_key_iv: "6a2b610a188e6f9b038a0fbdd7ab7547"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.962803 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "eed1081d2c3e466393ce04310c6f5579"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "fb976d0540dcaf1f09f8090f98ece94e"
server_key_iv: "6a2b610a188e6f9b038a0fbdd7ab7547"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.964108 24822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:40:22.964833 24837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:22.965000 24822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:22.965078 24822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "eed1081d2c3e466393ce04310c6f5579"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "fb976d0540dcaf1f09f8090f98ece94e"
server_key_iv: "6a2b610a188e6f9b038a0fbdd7ab7547"
server_key_version: "encryptionkey@0"
I20240506 15:40:22.965145 24822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:22.980145 24822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:22.982342 24822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:22.982472 24822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:22.982721 24822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:22.983080 24822 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:22.983122 24822 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:22.983155 24822 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:22.983181 24822 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:22.988739 24822 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:41639
I20240506 15:40:22.988794 24950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:41639 every 8 connection(s)
I20240506 15:40:22.989089 24822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:40:22.991145 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 24822
I20240506 15:40:22.991235 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:40:22.991390 18371 external_mini_cluster.cc:1428] Setting key d1bd472f6af6853523d22325b2c6c364
May 06 15:40:22 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:22.997470 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:22.989763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:40669 (local address 127.17.240.254:43707)
0506 15:40:22.989817 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:22.989819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:22.989968 (+ 149us) server_negotiation.cc:407] Connection header received
0506 15:40:22.990495 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:22.990496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:22.990507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:22.990578 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:22.991807 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.992513 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:22.993029 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.993135 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:22.995057 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:22.995062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:22.995062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:22.995070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:22.996464 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:22.996809 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:22.996810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:22.996810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:22.996826 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:22.996972 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:22.996972 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:22.996972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:22.997058 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:22.997103 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:22.997329 (+ 226us) server_negotiation.cc:299] Negotiation successful
0506 15:40:22.997399 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240506 15:40:22.997586 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:22.989718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:40669)
0506 15:40:22.989863 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:22.989893 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:22.990346 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:22.990635 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:22.990639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:22.990944 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:22.991719 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:22.991726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.992592 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:22.992593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:22.992963 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:22.992964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:22.993072 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:22.993589 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:22.993602 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:22.994934 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:22.996545 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:22.996549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:22.996552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:22.996747 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:22.996878 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:22.996878 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:22.996879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:22.996929 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:22.997109 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:22.997111 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:22.997227 (+ 116us) client_negotiation.cc:769] Sending connection context
0506 15:40:22.997313 (+ 86us) client_negotiation.cc:240] Negotiation successful
0506 15:40:22.997417 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20240506 15:40:22.998172 24951 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43707
I20240506 15:40:22.998287 24951 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:22.998509 24951 heartbeater.cc:511] Master 127.17.240.254:43707 requested a full tablet report, sending...
I20240506 15:40:22.999029 24492 ts_manager.cc:194] Registered new tserver with Master: eed1081d2c3e466393ce04310c6f5579 (127.17.240.195:41639)
I20240506 15:40:22.999553 24492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:40669
I20240506 15:40:23.003557 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.240.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.253.keytab.
Entry for principal oryx/127.17.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.253.keytab.
I20240506 15:40:23.031145 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.253:38535
--webserver_interface=127.17.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.253.keytab
--principal=oryx/127.17.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:43707,127.17.240.253:38535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:23.113574 24956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:23.113785 24956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:23.113808 24956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:23.115111 24956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:23.115159 24956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:23.115172 24956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:23.115185 24956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:23.115195 24956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:40:23.116688 24956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:43707,127.17.240.253:38535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.253.keytab
--principal=oryx/127.17.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.253:38535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
W20240506 15:40:23.116789 24956 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.17.240.254:43707,127.17.240.253:38535'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20240506 15:40:23.116927 24956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:23.117120 24956 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 15:40:23.122781 24956 server_base.cc:1034] running on GCE node
W20240506 15:40:23.123720 24964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.124593 24961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.125385 24962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:23.125670 24956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:23.125926 24956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:23.127086 24956 hybrid_clock.cc:648] HybridClock initialized: now 1715010023127072 us; error 32 us; skew 500 ppm
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:23.129751 24956 init.cc:377] Logged in from keytab as oryx/127.17.240.253@KRBTEST.COM (short username oryx)
I20240506 15:40:23.130203 24956 webserver.cc:460] Webserver started at http://127.17.240.253:43567/ using document root <none> and password file <none>
I20240506 15:40:23.130399 24956 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:23.130447 24956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:23.130560 24956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:23.131676 24956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data/instance:
uuid: "1167d7fcd2694c9488abfc0248fc57e1"
format_stamp: "Formatted at 2024-05-06 15:40:23 on dist-test-slave-v0k1"
server_key: "1378546cfb831a6d994283599b651dcf"
server_key_iv: "c586e74f3d25cc02983b13d486583794"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.132009 24956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal/instance:
uuid: "1167d7fcd2694c9488abfc0248fc57e1"
format_stamp: "Formatted at 2024-05-06 15:40:23 on dist-test-slave-v0k1"
server_key: "1378546cfb831a6d994283599b651dcf"
server_key_iv: "c586e74f3d25cc02983b13d486583794"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.133333 24956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:40:23.134197 24971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.134467 24956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:23.134568 24956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal
uuid: "1167d7fcd2694c9488abfc0248fc57e1"
format_stamp: "Formatted at 2024-05-06 15:40:23 on dist-test-slave-v0k1"
server_key: "1378546cfb831a6d994283599b651dcf"
server_key_iv: "c586e74f3d25cc02983b13d486583794"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.134644 24956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:23.165939 24956 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:23.168893 24956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:23.169052 24956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:23.174077 24956 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.253:38535
I20240506 15:40:23.174176 25023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.253:38535 every 8 connection(s)
I20240506 15:40:23.174542 24956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
I20240506 15:40:23.175637 25024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:23.176348 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 24956
I20240506 15:40:23.177816 25024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.253@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.240.253@KRBTEST.COM
I20240506 15:40:23.186214 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.178554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.17.240.254:43707)
0506 15:40:23.178601 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.178603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.178718 (+ 115us) server_negotiation.cc:407] Connection header received
0506 15:40:23.178874 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.178875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.178887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.178955 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.179650 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.180166 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.180799 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.180993 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.183183 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.183188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.183188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.183196 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.184860 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.185305 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.185306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.185307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.185335 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.185509 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.185510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.185510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.185646 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.185711 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.185979 (+ 268us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.186114 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240506 15:40:23.186308 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.178459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:37516)
0506 15:40:23.178601 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.178631 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.178778 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.178971 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.178973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.179316 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.179544 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.179548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.180244 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.180245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.180707 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.180709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.180813 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.181420 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.181437 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.183091 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.184964 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.184970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.184973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.185232 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.185392 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.185393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.185393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.185454 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.185708 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.185714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.185846 (+ 132us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.185947 (+ 101us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.186105 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20240506 15:40:23.186686 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.177890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.17.240.253:38535)
0506 15:40:23.178105 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.178119 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.178168 (+ 49us) server_negotiation.cc:407] Connection header received
0506 15:40:23.178778 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.178786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.178810 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.178896 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.179667 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.180388 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.181122 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.181340 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.183713 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.183726 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.183726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.183731 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.185356 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.185680 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.185684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.185685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.185712 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.185852 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.185853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.185853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.185955 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.186094 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.186179 (+ 85us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.186617 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20240506 15:40:23.186915 25024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.253@KRBTEST.COM for oryx/127.17.240.253@KRBTEST.COM
I20240506 15:40:23.193601 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.187165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38535 (local address 127.0.0.1:58400)
0506 15:40:23.187276 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.187286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.187335 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.187454 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.187455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.187670 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.187878 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.187880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.188594 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.188595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.188967 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.188968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.189046 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.189538 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.189541 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.191089 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.192740 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.192741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.192741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.192884 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.193046 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.193047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.193048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.193076 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.193324 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.193325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.193364 (+ 39us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.193429 (+ 65us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.193510 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20240506 15:40:23.193692 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.187198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.17.240.253:38535)
0506 15:40:23.187235 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.187237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.187335 (+ 98us) server_negotiation.cc:407] Connection header received
0506 15:40:23.187390 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.187391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.187400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.187451 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.187964 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.188513 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.189201 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.189320 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.191185 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.191188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.191188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.191196 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.192645 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.192951 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.192952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.192953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.192968 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.193160 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.193161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.193162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.193254 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.193318 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.193481 (+ 163us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.193573 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":18}
I20240506 15:40:23.195304 25024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: Bootstrap starting.
I20240506 15:40:23.195824 25024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:23.196046 25024 log.cc:824] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:23.196755 25024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: No bootstrap required, opened a new log
I20240506 15:40:23.198006 25024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } }
I20240506 15:40:23.198133 25024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:23.198153 25024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1167d7fcd2694c9488abfc0248fc57e1, State: Initialized, Role: FOLLOWER
I20240506 15:40:23.198246 25024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } }
I20240506 15:40:23.198524 25024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:23.198489 25030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } } }
I20240506 15:40:23.198587 25030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: This master's current role is: FOLLOWER
W20240506 15:40:23.200279 25041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:40:23.200335 25041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:40:23.202026 24956 master_runner.cc:428] Detected that this master 1167d7fcd2694c9488abfc0248fc57e1 is joining an existing cluster
I20240506 15:40:23.202075 24956 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.240.253:38535
I20240506 15:40:23.209692 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.203321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.17.240.253:38535)
0506 15:40:23.203353 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.203355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.203456 (+ 101us) server_negotiation.cc:407] Connection header received
0506 15:40:23.203530 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.203531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.203540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.203583 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.204383 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.205079 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.205706 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.205881 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.206741 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.206744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.206745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.206750 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.208646 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.208995 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.208996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.208996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.209016 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.209159 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.209160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.209160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.209271 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.209340 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.209477 (+ 137us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.209611 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":19}
I20240506 15:40:23.209692 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.203202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38535 (local address 127.0.0.1:58416)
0506 15:40:23.203403 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.203412 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.203473 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.203628 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.203629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.203966 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.204300 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.204302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.205187 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.205189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.205628 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.205631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.205709 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.206315 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.206319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.206663 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.208733 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.208734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.208734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.208930 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.209082 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.209083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.209083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.209109 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.209330 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.209331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.209368 (+ 37us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.209448 (+ 80us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.209561 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":105,"threads_started":1}
I20240506 15:40:23.243530 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.203167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:37520)
0506 15:40:23.203345 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.203357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.203438 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.203606 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.203607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.203968 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.204300 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.204302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.206537 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.206538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.207169 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.207171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.207247 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.239680 (+ 32433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.239686 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.240125 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.242318 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.242320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.242321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.242572 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.242762 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.242763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.242764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.242803 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.243118 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.243119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.243177 (+ 58us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.243271 (+ 94us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.243399 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20240506 15:40:23.243541 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.203235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.17.240.254:43707)
0506 15:40:23.203284 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.203286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.203446 (+ 160us) server_negotiation.cc:407] Connection header received
0506 15:40:23.203499 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.203501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.203516 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.203597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.205197 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.206430 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.207246 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.207393 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.240274 (+ 32881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.240282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.240282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.240292 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.242205 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.242671 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.242672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.242672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.242698 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.242866 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.242867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.242867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.242995 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.243085 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.243329 (+ 244us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.243427 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for oryx/127.17.240.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.252@KRBTEST.COM" created.
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.254@KRBTEST.COM for oryx/127.17.240.253@KRBTEST.COM
I20240506 15:40:23.253921 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.246599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38535 (local address 127.0.0.1:58428)
0506 15:40:23.246763 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.246794 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.246876 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.247068 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.247071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.247437 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.247610 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.247613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.248199 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.248200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.248547 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.248548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.248615 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.249219 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.249224 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.250623 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.252635 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.252643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.252646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.252943 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.253140 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.253141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.253142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.253179 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.253473 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.253480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.253539 (+ 59us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.253612 (+ 73us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.253750 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20240506 15:40:23.254627 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.246712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58428 (local address 127.17.240.253:38535)
0506 15:40:23.246759 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.246760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.246859 (+ 99us) server_negotiation.cc:407] Connection header received
0506 15:40:23.246961 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.246962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.246972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.247060 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.247701 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.248123 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.248618 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.248797 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.250719 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.250722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.250722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.250729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.252527 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.253040 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.253041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.253042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.253070 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.253278 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.253279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.253279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.253395 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.254348 (+ 953us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.254397 (+ 49us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.254493 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240506 15:40:23.255513 24492 catalog_manager.cc:6968] Initiating ChangeConfig request to add master 127.17.240.253:38535: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1"
member_type: NON_VOTER
last_known_addr {
host: "127.17.240.253"
port: 38535
}
attrs {
promote: true
}
}
dest_uuid: "0a137b87b00841f5a73fc1d4078fc43c"
cas_config_opid_index: -1
I20240506 15:40:23.255853 24492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } }
I20240506 15:40:23.256650 24998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:23.256661 25053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1167d7fcd2694c9488abfc0248fc57e1 (127.17.240.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } } }
I20240506 15:40:23.257624 24998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Refusing update from remote peer 0a137b87b00841f5a73fc1d4078fc43c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240506 15:40:23.257896 25053 catalog_manager.cc:6972] Successfully completed master ChangeConfig request to add master 127.17.240.253:38535
I20240506 15:40:23.258095 25053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:23.258162 24956 master.cc:537] Master@127.17.240.253:38535 shutting down...
I20240506 15:40:23.258344 25054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a137b87b00841f5a73fc1d4078fc43c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } } }
I20240506 15:40:23.258425 25054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: This master's current role is: LEADER
W20240506 15:40:23.258855 24479 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20240506 15:40:23.259372 24479 consensus_peers.cc:485] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c -> Peer 1167d7fcd2694c9488abfc0248fc57e1 (127.17.240.253:38535): Couldn't send request to peer 1167d7fcd2694c9488abfc0248fc57e1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:40:23.261041 24956 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:23.261127 24956 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:23.261168 24956 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.252.keytab.
Entry for principal oryx/127.17.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.252.keytab.
I20240506 15:40:23.266882 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.252:44923
--webserver_interface=127.17.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.252.keytab
--principal=oryx/127.17.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:43707,127.17.240.253:38535,127.17.240.252:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:40:23.273465 24956 master.cc:552] Master@127.17.240.253:38535 shutdown complete.
I20240506 15:40:23.273576 24956 master_runner.cc:305] Clearing existing system tablet
I20240506 15:40:23.273739 24956 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:23.273789 24956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:23.274564 24956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20240506 15:40:23.275152 25060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.275280 24956 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:23.275336 24956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:23.275362 24956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal
uuid: "1167d7fcd2694c9488abfc0248fc57e1"
format_stamp: "Formatted at 2024-05-06 15:40:23 on dist-test-slave-v0k1"
server_key: "1378546cfb831a6d994283599b651dcf"
server_key_iv: "c586e74f3d25cc02983b13d486583794"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.276233 24956 ts_tablet_manager.cc:1875] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:23.278153 24956 ts_tablet_manager.cc:1888] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20240506 15:40:23.278232 24956 log.cc:1196] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20240506 15:40:23.278477 24956 ts_tablet_manager.cc:1909] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: Deleting consensus metadata
I20240506 15:40:23.278614 24956 master_runner.cc:315] Copying system tablet from 127.17.240.254:43707
I20240506 15:40:23.279335 24956 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.240.254:43707
I20240506 15:40:23.286453 25066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.279678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:37536)
0506 15:40:23.279823 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.279833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.279898 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.280070 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.280071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.280291 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.280564 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.280567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.281556 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.281557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.282180 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.282183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.282284 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.282992 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.282996 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.283409 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.285415 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.285416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.285417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.285622 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.285794 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.285794 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.285795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.285827 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.286076 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.286077 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.286133 (+ 56us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.286207 (+ 74us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.286334 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20240506 15:40:23.286480 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.279779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.17.240.254:43707)
0506 15:40:23.279829 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.279832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.279882 (+ 50us) server_negotiation.cc:407] Connection header received
0506 15:40:23.279958 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.279959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.279970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.280049 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.280673 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.281438 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.282290 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.282460 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.283509 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.283515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.283515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.283523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.285313 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.285706 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.285707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.285707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.285732 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.285887 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.285888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.285888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.286014 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.286080 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.286244 (+ 164us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.286389 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240506 15:40:23.287447 24512 tablet_copy_service.cc:140] P 0a137b87b00841f5a73fc1d4078fc43c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1167d7fcd2694c9488abfc0248fc57e1 ({username='oryx', principal='oryx/127.17.240.253@KRBTEST.COM'} at 127.0.0.1:37536)
I20240506 15:40:23.287554 24512 tablet_copy_service.cc:161] P 0a137b87b00841f5a73fc1d4078fc43c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1167d7fcd2694c9488abfc0248fc57e1 at {username='oryx', principal='oryx/127.17.240.253@KRBTEST.COM'} at 127.0.0.1:37536: session id = 1167d7fcd2694c9488abfc0248fc57e1-00000000000000000000000000000000
I20240506 15:40:23.288293 24512 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:40:23.288895 24956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20240506 15:40:23.290439 24956 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: tablet copy: Starting download of 0 data blocks...
I20240506 15:40:23.290685 24956 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: tablet copy: Starting download of 1 WAL segments...
I20240506 15:40:23.291803 24956 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:40:23.292910 24512 tablet_copy_service.cc:342] P 0a137b87b00841f5a73fc1d4078fc43c: Request end of tablet copy session 1167d7fcd2694c9488abfc0248fc57e1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.240.253@KRBTEST.COM'} at 127.0.0.1:37536
I20240506 15:40:23.292984 24512 tablet_copy_service.cc:433] P 0a137b87b00841f5a73fc1d4078fc43c: ending tablet copy session 1167d7fcd2694c9488abfc0248fc57e1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1167d7fcd2694c9488abfc0248fc57e1
I20240506 15:40:23.296065 24956 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:23.298468 25070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.299257 25067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.300081 25068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:23.300818 24956 server_base.cc:1034] running on GCE node
I20240506 15:40:23.300931 24956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:23.301095 24956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:23.302203 24956 hybrid_clock.cc:648] HybridClock initialized: now 1715010023302197 us; error 29 us; skew 500 ppm
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:23.304010 24956 init.cc:377] Logged in from keytab as oryx/127.17.240.253@KRBTEST.COM (short username oryx)
I20240506 15:40:23.304322 24956 webserver.cc:460] Webserver started at http://127.17.240.253:42073/ using document root <none> and password file <none>
I20240506 15:40:23.304451 24956 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:23.304482 24956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:23.305223 24956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:23.305806 25077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.305914 24956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.305951 24956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal
uuid: "1167d7fcd2694c9488abfc0248fc57e1"
format_stamp: "Formatted at 2024-05-06 15:40:23 on dist-test-slave-v0k1"
server_key: "1378546cfb831a6d994283599b651dcf"
server_key_iv: "c586e74f3d25cc02983b13d486583794"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.306010 24956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:23.338400 24956 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:23.340524 24956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:23.343647 24956 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.253:38535
I20240506 15:40:23.344007 24956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
I20240506 15:40:23.347136 25129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.253:38535 every 8 connection(s)
I20240506 15:40:23.348117 25130 sys_catalog.cc:263] Verifying existing consensus state
I20240506 15:40:23.348712 25130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: Bootstrap starting.
I20240506 15:40:23.351867 25130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:40:23.352098 25130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: Bootstrap complete.
I20240506 15:40:23.352314 25130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } }
I20240506 15:40:23.352444 25130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1167d7fcd2694c9488abfc0248fc57e1, State: Initialized, Role: LEARNER
I20240506 15:40:23.352483 25130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } }
I20240506 15:40:23.352694 25133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } } }
I20240506 15:40:23.352842 25133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: This master's current role is: LEARNER
I20240506 15:40:23.352710 25130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:23.354880 25144 catalog_manager.cc:1259] Loaded cluster ID: 2798417bad99443ebbf226345980009d
I20240506 15:40:23.354977 25144 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: loading cluster ID for follower catalog manager: success
I20240506 15:40:23.356302 25144 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: acquiring CA information for follower catalog manager: success
I20240506 15:40:23.356644 25144 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240506 15:40:23.368829 25058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:23.369066 25058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:23.369092 25058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:23.370540 25058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:23.370591 25058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:23.370605 25058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:23.370618 25058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:23.370647 25058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:40:23.372334 25058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:43707,127.17.240.253:38535,127.17.240.252:44923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.252.keytab
--principal=oryx/127.17.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.252:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:23.372594 25058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:23.372795 25058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:23.378108 25149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.378808 25152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.379588 25150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:23.380338 25058 server_base.cc:1034] running on GCE node
I20240506 15:40:23.380620 25058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:23.380892 25058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:23.382053 25058 hybrid_clock.cc:648] HybridClock initialized: now 1715010023382039 us; error 37 us; skew 500 ppm
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:23.384785 25058 init.cc:377] Logged in from keytab as oryx/127.17.240.252@KRBTEST.COM (short username oryx)
I20240506 15:40:23.385185 25058 webserver.cc:460] Webserver started at http://127.17.240.252:38819/ using document root <none> and password file <none>
I20240506 15:40:23.385357 25058 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:23.385394 25058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:23.385480 25058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:23.386512 25058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data/instance:
uuid: "a512f3f825a341bb99bd83daa7178abe"
format_stamp: "Formatted at 2024-05-06 15:40:23 on dist-test-slave-v0k1"
server_key: "b520fa7d79ae0773f34fa5a3facab6b0"
server_key_iv: "03c668ac2e96e0d50f28f10bb34c7425"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.386831 25058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal/instance:
uuid: "a512f3f825a341bb99bd83daa7178abe"
format_stamp: "Formatted at 2024-05-06 15:40:23 on dist-test-slave-v0k1"
server_key: "b520fa7d79ae0773f34fa5a3facab6b0"
server_key_iv: "03c668ac2e96e0d50f28f10bb34c7425"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.388127 25058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:40:23.388833 25159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.389029 25058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:40:23.389117 25058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal
uuid: "a512f3f825a341bb99bd83daa7178abe"
format_stamp: "Formatted at 2024-05-06 15:40:23 on dist-test-slave-v0k1"
server_key: "b520fa7d79ae0773f34fa5a3facab6b0"
server_key_iv: "03c668ac2e96e0d50f28f10bb34c7425"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.389186 25058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:23.404232 25058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:23.406626 25058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:23.406764 25058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:23.411093 25058 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.252:44923
I20240506 15:40:23.411136 25211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.252:44923 every 8 connection(s)
I20240506 15:40:23.411459 25058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
I20240506 15:40:23.412330 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 25058
I20240506 15:40:23.412361 25212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:23.414527 25212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.240.252@KRBTEST.COM
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.252@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:23.421831 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.415246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.17.240.254:43707)
0506 15:40:23.415317 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.415320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.415331 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:40:23.415359 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.415360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.415373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.415433 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.416062 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.416551 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.417094 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.417293 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.419343 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.419347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.419348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.419355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.420719 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.421097 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.421098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.421099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.421124 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.421263 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.421263 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.421264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.421376 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.421437 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.421633 (+ 196us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.421711 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240506 15:40:23.421929 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.413155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.17.240.252:44923)
0506 15:40:23.413361 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.413372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.413398 (+ 26us) server_negotiation.cc:407] Connection header received
0506 15:40:23.413860 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.413868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.413924 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.414091 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.414792 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.415806 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.416353 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.416589 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.418820 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.418830 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.418830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.418839 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.420540 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.420854 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.420857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.420857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.420897 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.421052 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.421053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.421053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.421317 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.421466 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.421541 (+ 75us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.421677 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20240506 15:40:23.421931 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.415078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:37538)
0506 15:40:23.415229 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.415258 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.415319 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.415447 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.415449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.415778 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.415966 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.415967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.416632 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.416634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.417003 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.417005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.417097 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.417632 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.417646 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.419256 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.420789 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.420794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.420797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.420995 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.421190 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.421190 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.421191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.421214 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.421433 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.421438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.421504 (+ 66us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.421595 (+ 91us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.421745 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":33,"thread_start_us":80,"threads_started":1}
I20240506 15:40:23.422551 25212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.252@KRBTEST.COM for oryx/127.17.240.253@KRBTEST.COM
I20240506 15:40:23.428896 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.422860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.17.240.253:38535)
0506 15:40:23.422992 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.422994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.423003 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:40:23.423051 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.423051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.423063 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.423124 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.423537 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.424207 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.424727 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.424839 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.426404 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.426409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.426410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.426415 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.427842 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.428171 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.428173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.428173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.428201 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.428370 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.428371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.428372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.428480 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.428550 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.428741 (+ 191us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.428824 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20240506 15:40:23.428916 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.422786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38535 (local address 127.0.0.1:58434)
0506 15:40:23.422832 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.422841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.422873 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.423134 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.423134 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.423335 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.423465 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.423466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.424280 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.424280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.424664 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.424666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.424725 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.425184 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.425188 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.426321 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.427923 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.427924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.427924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.428061 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.428270 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.428271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.428271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.428296 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.428586 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.428587 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.428631 (+ 44us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.428700 (+ 69us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.428819 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20240506 15:40:23.429394 25212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.252" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.252@KRBTEST.COM for oryx/127.17.240.252@KRBTEST.COM
I20240506 15:40:23.435180 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.429653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:44923 (local address 127.0.0.1:52012)
0506 15:40:23.429703 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.429713 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.429755 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.429853 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.429854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.429961 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.430082 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.430083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.430796 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.430797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.431225 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.431226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.431286 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.431749 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.431753 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.432851 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.434354 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.434355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.434356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.434498 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.434678 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.434679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.434679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.434707 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.434893 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.434894 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.434925 (+ 31us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.434973 (+ 48us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.435113 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20240506 15:40:23.435250 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.429688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.17.240.252:44923)
0506 15:40:23.429721 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.429723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.429744 (+ 21us) server_negotiation.cc:407] Connection header received
0506 15:40:23.429790 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.429790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.429797 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.429845 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.430159 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.430600 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.431307 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.431466 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.432925 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.432928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.432928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.432934 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.434269 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.434590 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.434591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.434591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.434606 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.434770 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.434771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.434771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.434849 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.434892 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.434993 (+ 101us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.435169 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":17}
I20240506 15:40:23.436744 25212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: Bootstrap starting.
I20240506 15:40:23.437289 25212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:23.437525 25212 log.cc:824] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:23.438269 25212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: No bootstrap required, opened a new log
I20240506 15:40:23.439698 25212 raft_consensus.cc:348] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 44923 } }
I20240506 15:40:23.439810 25212 raft_consensus.cc:374] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:23.439826 25212 raft_consensus.cc:729] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a512f3f825a341bb99bd83daa7178abe, State: Initialized, Role: FOLLOWER
I20240506 15:40:23.439931 25212 consensus_queue.cc:260] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 44923 } }
I20240506 15:40:23.440208 25212 sys_catalog.cc:564] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:23.440191 25218 sys_catalog.cc:455] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 44923 } } }
I20240506 15:40:23.440325 25218 sys_catalog.cc:458] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [sys.catalog]: This master's current role is: FOLLOWER
W20240506 15:40:23.441898 25229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:40:23.441984 25229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:40:23.443943 25058 master_runner.cc:428] Detected that this master a512f3f825a341bb99bd83daa7178abe is joining an existing cluster
I20240506 15:40:23.443981 25058 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.240.252:44923
I20240506 15:40:23.473608 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 24554
I20240506 15:40:23.480909 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.445327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.17.240.254:43707)
0506 15:40:23.445375 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.445377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.445384 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:40:23.445404 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.445405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.445416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.445474 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.471998 (+ 26524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.472770 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.474408 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.474615 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.477155 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.477160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.477160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.477169 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.479827 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.480221 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.480222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.480222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.480249 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.480408 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.480409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.480409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.480514 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.480554 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.480751 (+ 197us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.480828 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240506 15:40:23.480904 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.445308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.17.240.252:44923)
0506 15:40:23.445344 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.445346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.445483 (+ 137us) server_negotiation.cc:407] Connection header received
0506 15:40:23.445587 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.445587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.445595 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.445646 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.472014 (+ 26368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.472728 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.473574 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.473753 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.477072 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.477076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.477076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.477083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.479746 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.480174 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.480176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.480176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.480197 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.480346 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.480346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.480347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.480481 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.480533 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.480714 (+ 181us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.480801 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:40:23.481231 25235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.445152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:44923 (local address 127.0.0.1:52020)
0506 15:40:23.445425 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.445434 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.445503 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.445650 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.445651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.471465 (+ 25814us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.471889 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.471892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.472842 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.472843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.473449 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.473452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.473803 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.475198 (+ 1395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.475203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.476983 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.479853 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.479854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.479854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.480098 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.480256 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.480256 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.480257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.480286 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.480575 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.480576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.480611 (+ 35us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.481000 (+ 389us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.481108 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":66,"threads_started":1}
I20240506 15:40:23.481724 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.444939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:37542)
0506 15:40:23.445097 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.445107 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.445161 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.445534 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.445535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.471463 (+ 25928us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.471834 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.471839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.472907 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.472908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.474301 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.474304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.474666 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.475179 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.475184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.475780 (+ 596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.479910 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.479911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.479911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.480136 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.480315 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.480316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.480316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.480346 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.480596 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.480596 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.480629 (+ 33us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.481525 (+ 896us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.481634 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20240506 15:40:23.482259 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:36253
--local_ip_for_outbound_sockets=127.17.240.193
--tserver_master_addrs=127.17.240.254:43707
--webserver_port=46801
--webserver_interface=127.17.240.193
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab
--principal=oryx/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.240.254:43707,127.17.240.253:38535
--tserver_master_addrs=127.17.240.254:43707,127.17.240.253:38535,127.17.240.252:44923 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:40:23.486868 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.445009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.17.240.253:38535)
0506 15:40:23.445081 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.445083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.445305 (+ 222us) server_negotiation.cc:407] Connection header received
0506 15:40:23.445401 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.445402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.445413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.445443 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.471997 (+ 26554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.472828 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.482456 (+ 9628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.482672 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.483918 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.483922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.483923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.483930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.485752 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.486114 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.486116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.486116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.486150 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.486319 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.486320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.486320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.486429 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.486470 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.486671 (+ 201us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.486771 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240506 15:40:23.487396 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.444954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38535 (local address 127.0.0.1:58446)
0506 15:40:23.445250 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.445257 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.445328 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.445515 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.445516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.471458 (+ 25942us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.471834 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.471839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.473059 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.473059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.482338 (+ 9279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.482342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.482738 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.483435 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.483440 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.483807 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.485831 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.485832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.485832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.486001 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.486223 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.486224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.486224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.486252 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.486511 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.486511 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.486555 (+ 44us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.487184 (+ 629us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.487288 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":77,"threads_started":1}
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010022, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.254@KRBTEST.COM for oryx/127.17.240.252@KRBTEST.COM
I20240506 15:40:23.492348 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.485425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52028 (local address 127.17.240.252:44923)
0506 15:40:23.485463 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.485464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.485935 (+ 471us) server_negotiation.cc:407] Connection header received
0506 15:40:23.486009 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.486010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.486021 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.486072 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.486703 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.487378 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.488098 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.488284 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.490022 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.490027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.490027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.490032 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.491357 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.491710 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.491710 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.491711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.491729 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.491864 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.491865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.491865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.491948 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.492026 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.492179 (+ 153us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.492255 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240506 15:40:23.492350 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.485381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:44923 (local address 127.0.0.1:52028)
0506 15:40:23.485868 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.485879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.485930 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.486136 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.486137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.486332 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.486578 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.486582 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.487553 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.487554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.488025 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.488027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.488093 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.488733 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.488737 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.489933 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.491439 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.491440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.491440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.491639 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.491782 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.491783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.491783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.491811 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.492054 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.492054 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.492092 (+ 38us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.492150 (+ 58us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.492256 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466}
I20240506 15:40:23.493124 24492 catalog_manager.cc:6968] Initiating ChangeConfig request to add master 127.17.240.252:44923: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a512f3f825a341bb99bd83daa7178abe"
member_type: NON_VOTER
last_known_addr {
host: "127.17.240.252"
port: 44923
}
attrs {
promote: true
}
}
dest_uuid: "0a137b87b00841f5a73fc1d4078fc43c"
cas_config_opid_index: 5
I20240506 15:40:23.493451 24492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } }
I20240506 15:40:23.494012 25055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a512f3f825a341bb99bd83daa7178abe (127.17.240.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.494861 25055 catalog_manager.cc:6972] Successfully completed master ChangeConfig request to add master 127.17.240.252:44923
I20240506 15:40:23.494932 25055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a137b87b00841f5a73fc1d4078fc43c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.494982 25055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:23.495395 25186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:23.496428 25186 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [term 1 FOLLOWER]: Refusing update from remote peer 0a137b87b00841f5a73fc1d4078fc43c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240506 15:40:23.496574 25058 master.cc:537] Master@127.17.240.252:44923 shutting down...
I20240506 15:40:23.497552 25053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:23.498371 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.493954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38535 (local address 127.0.0.1:58458)
0506 15:40:23.495483 (+ 1529us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.495495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.495567 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.495866 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.495867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.495869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:23.496069 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.496071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.497038 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.497040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.498097 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.498100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.498163 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.498215 (+ 52us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.498248 (+ 33us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.498299 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1510}
W20240506 15:40:23.498426 24477 consensus_peers.cc:485] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c -> Peer a512f3f825a341bb99bd83daa7178abe (127.17.240.252:44923): Couldn't send request to peer a512f3f825a341bb99bd83daa7178abe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:40:23.499204 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.494017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.17.240.253:38535)
0506 15:40:23.494058 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.494060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.495666 (+ 1606us) server_negotiation.cc:407] Connection header received
0506 15:40:23.495696 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.495697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.495728 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.495790 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:23.496160 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.496911 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.498648 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.499027 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.499106 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.499126 (+ 20us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.499158 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:40:23.499593 25104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 LEARNER]: Refusing update from remote peer 0a137b87b00841f5a73fc1d4078fc43c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240506 15:40:23.499833 25053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:40:23.500392 25104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a512f3f825a341bb99bd83daa7178abe (127.17.240.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.501507 25240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a137b87b00841f5a73fc1d4078fc43c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.501624 25240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: This master's current role is: LEARNER
I20240506 15:40:23.501724 25053 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c: attempting to promote NON_VOTER 1167d7fcd2694c9488abfc0248fc57e1 to VOTER
I20240506 15:40:23.501902 25053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } }
I20240506 15:40:23.502383 25104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 LEARNER]: Refusing update from remote peer 0a137b87b00841f5a73fc1d4078fc43c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240506 15:40:23.502496 25058 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:23.502614 25058 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:23.502676 25058 tablet_replica.cc:330] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: stopping tablet replica
W20240506 15:40:23.502846 24477 consensus_peers.cc:485] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c -> Peer a512f3f825a341bb99bd83daa7178abe (127.17.240.252:44923): Couldn't send request to peer a512f3f825a341bb99bd83daa7178abe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:40:23.501525 25133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a137b87b00841f5a73fc1d4078fc43c. Latest consensus state: current_term: 1 leader_uuid: "0a137b87b00841f5a73fc1d4078fc43c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: NON_VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: true } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.502972 25133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: This master's current role is: LEARNER
I20240506 15:40:23.503067 25052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240506 15:40:23.503850 25052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1167d7fcd2694c9488abfc0248fc57e1 (127.17.240.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.504705 25052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a137b87b00841f5a73fc1d4078fc43c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.504779 25052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:23.504833 25052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a137b87b00841f5a73fc1d4078fc43c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.504873 25052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:23.505398 25104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1167d7fcd2694c9488abfc0248fc57e1 (127.17.240.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.506410 25133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a137b87b00841f5a73fc1d4078fc43c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.506501 25133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:40:23.517900 25058 master.cc:552] Master@127.17.240.252:44923 shutdown complete.
I20240506 15:40:23.517956 25058 master_runner.cc:305] Clearing existing system tablet
I20240506 15:40:23.518107 25058 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:23.518157 25058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:23.518790 25058 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.519351 25242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.519450 25058 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:23.519495 25058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.519524 25058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal
uuid: "a512f3f825a341bb99bd83daa7178abe"
format_stamp: "Formatted at 2024-05-06 15:40:23 on dist-test-slave-v0k1"
server_key: "b520fa7d79ae0773f34fa5a3facab6b0"
server_key_iv: "03c668ac2e96e0d50f28f10bb34c7425"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.520121 25058 ts_tablet_manager.cc:1875] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:23.521656 25058 ts_tablet_manager.cc:1888] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20240506 15:40:23.521725 25058 log.cc:1196] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20240506 15:40:23.521921 25058 ts_tablet_manager.cc:1909] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: Deleting consensus metadata
I20240506 15:40:23.522040 25058 master_runner.cc:315] Copying system tablet from 127.17.240.254:43707
I20240506 15:40:23.522643 25058 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: tablet copy: Beginning tablet copy session from remote peer at address 127.17.240.254:43707
I20240506 15:40:23.528259 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.523045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.17.240.254:43707)
0506 15:40:23.523121 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.523123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.523198 (+ 75us) server_negotiation.cc:407] Connection header received
0506 15:40:23.523277 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.523277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.523289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.523351 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.523820 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.524306 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.524917 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.525472 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.525836 (+ 364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.525841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.525841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.525848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.527288 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.527633 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.527634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.527634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.527656 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.527795 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.527795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.527796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.527883 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.527940 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.528083 (+ 143us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.528161 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240506 15:40:23.528281 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.522970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:37558)
0506 15:40:23.523139 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.523149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.523214 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.523362 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.523362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.523530 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.523730 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.523732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.524379 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.524381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.524834 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.524836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.524902 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.525420 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.525425 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.525759 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.527366 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.527367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.527368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.527515 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.527712 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.527712 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.527713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.527742 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.527934 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.527935 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.527982 (+ 47us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.528049 (+ 67us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.528161 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20240506 15:40:23.528810 24512 tablet_copy_service.cc:140] P 0a137b87b00841f5a73fc1d4078fc43c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a512f3f825a341bb99bd83daa7178abe ({username='oryx', principal='oryx/127.17.240.252@KRBTEST.COM'} at 127.0.0.1:37558)
I20240506 15:40:23.528873 24512 tablet_copy_service.cc:161] P 0a137b87b00841f5a73fc1d4078fc43c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a512f3f825a341bb99bd83daa7178abe at {username='oryx', principal='oryx/127.17.240.252@KRBTEST.COM'} at 127.0.0.1:37558: session id = a512f3f825a341bb99bd83daa7178abe-00000000000000000000000000000000
I20240506 15:40:23.529353 24512 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:40:23.529817 25058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20240506 15:40:23.530778 25058 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: tablet copy: Starting download of 0 data blocks...
I20240506 15:40:23.530912 25058 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: tablet copy: Starting download of 1 WAL segments...
I20240506 15:40:23.531857 25058 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:40:23.532716 24512 tablet_copy_service.cc:342] P 0a137b87b00841f5a73fc1d4078fc43c: Request end of tablet copy session a512f3f825a341bb99bd83daa7178abe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.240.252@KRBTEST.COM'} at 127.0.0.1:37558
I20240506 15:40:23.532780 24512 tablet_copy_service.cc:433] P 0a137b87b00841f5a73fc1d4078fc43c: ending tablet copy session a512f3f825a341bb99bd83daa7178abe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a512f3f825a341bb99bd83daa7178abe
I20240506 15:40:23.535431 25058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:23.537679 25249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.538439 25252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.539113 25250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:23.539824 25058 server_base.cc:1034] running on GCE node
I20240506 15:40:23.539934 25058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:23.540107 25058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:23.541249 25058 hybrid_clock.cc:648] HybridClock initialized: now 1715010023541243 us; error 25 us; skew 500 ppm
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:23.542629 25058 init.cc:377] Logged in from keytab as oryx/127.17.240.252@KRBTEST.COM (short username oryx)
I20240506 15:40:23.542889 25058 webserver.cc:460] Webserver started at http://127.17.240.252:42073/ using document root <none> and password file <none>
I20240506 15:40:23.543040 25058 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:23.543079 25058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:23.543735 25058 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.544242 25259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.544368 25058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.544415 25058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal
uuid: "a512f3f825a341bb99bd83daa7178abe"
format_stamp: "Formatted at 2024-05-06 15:40:23 on dist-test-slave-v0k1"
server_key: "b520fa7d79ae0773f34fa5a3facab6b0"
server_key_iv: "03c668ac2e96e0d50f28f10bb34c7425"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.544466 25058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:23.554375 25058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:23.556131 25058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:23.559046 25058 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.252:44923
I20240506 15:40:23.559531 25058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
I20240506 15:40:23.571731 25312 sys_catalog.cc:263] Verifying existing consensus state
I20240506 15:40:23.572180 25312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: Bootstrap starting.
I20240506 15:40:23.573321 25311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.252:44923 every 8 connection(s)
I20240506 15:40:23.574718 25312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:40:23.574949 25312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: Bootstrap complete.
I20240506 15:40:23.575189 25312 raft_consensus.cc:348] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } }
I20240506 15:40:23.575264 25312 raft_consensus.cc:729] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a512f3f825a341bb99bd83daa7178abe, State: Initialized, Role: LEARNER
I20240506 15:40:23.575307 25312 consensus_queue.cc:260] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } }
I20240506 15:40:23.575536 25312 sys_catalog.cc:564] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:23.575536 25315 sys_catalog.cc:455] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } } }
I20240506 15:40:23.575714 25315 sys_catalog.cc:458] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe [sys.catalog]: This master's current role is: LEARNER
I20240506 15:40:23.577082 25326 catalog_manager.cc:1259] Loaded cluster ID: 2798417bad99443ebbf226345980009d
I20240506 15:40:23.577172 25326 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: loading cluster ID for follower catalog manager: success
I20240506 15:40:23.578406 25326 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: acquiring CA information for follower catalog manager: success
I20240506 15:40:23.578702 25326 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a512f3f825a341bb99bd83daa7178abe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240506 15:40:23.604705 25238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:23.605005 25238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:23.605051 25238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:23.607275 25238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:23.607357 25238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:23.607403 25238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:40:23.609910 25238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab
--principal=oryx/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:36253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=46801
--tserver_master_addrs=127.17.240.254:43707,127.17.240.253:38535,127.17.240.252:44923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:23.610172 25238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:23.610405 25238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:23.615916 25332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.616525 25331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.617316 25334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:23.618139 25238 server_base.cc:1034] running on GCE node
I20240506 15:40:23.618330 25238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:23.618551 25238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:23.619712 25238 hybrid_clock.cc:648] HybridClock initialized: now 1715010023619692 us; error 39 us; skew 500 ppm
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:23.628057 25238 init.cc:377] Logged in from keytab as oryx/127.17.240.193@KRBTEST.COM (short username oryx)
I20240506 15:40:23.628533 25238 webserver.cc:460] Webserver started at http://127.17.240.193:46801/ using document root <none> and password file <none>
I20240506 15:40:23.628732 25238 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:23.628787 25238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:23.630251 25238 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:23.630899 25341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.631086 25238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:23.631186 25238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "9899ac0aff6c40b1bdb86a2f5d922d76"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "91e15d8ffd3b430ef6afe54781fb2429"
server_key_iv: "e18ca85b4d4af716cf472f6b2c6ade09"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.631587 25238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:23.658028 25238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:23.660790 25238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:23.660933 25238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:23.661164 25238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:23.661523 25238 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:23.661561 25238 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.661583 25238 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:23.661595 25238 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.667570 25238 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:36253
I20240506 15:40:23.667670 25454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:36253 every 8 connection(s)
I20240506 15:40:23.667959 25238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:40:23.669674 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 25238
I20240506 15:40:23.669833 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 24688
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.253@KRBTEST.COM
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.252@KRBTEST.COM
I20240506 15:40:23.677621 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:38169
--local_ip_for_outbound_sockets=127.17.240.194
--tserver_master_addrs=127.17.240.254:43707
--webserver_port=41703
--webserver_interface=127.17.240.194
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab
--principal=oryx/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.240.254:43707,127.17.240.253:38535
--tserver_master_addrs=127.17.240.254:43707,127.17.240.253:38535,127.17.240.252:44923 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:40:23.680088 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.669176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:39311 (local address 127.17.240.254:43707)
0506 15:40:23.669237 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.669239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.669249 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:40:23.669591 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.669592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.669604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.669638 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.670919 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.671753 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.672551 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.672737 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.675898 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.675905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.675905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.675915 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.678267 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.678886 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.678888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.678888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.678917 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.679218 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.679219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.679220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.679367 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.679442 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.679827 (+ 385us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.679938 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240506 15:40:23.680171 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.668680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38535 (local address 127.17.240.193:42311)
0506 15:40:23.668815 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.668845 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.669457 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.669782 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.669785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.670514 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.670827 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.670829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.671858 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.671860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.672491 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.672494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.672861 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.673550 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.673575 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.676607 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.678792 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.678793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.678794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.678983 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.679205 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.679206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.679207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.679235 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.679565 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.679566 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.679655 (+ 89us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.679779 (+ 124us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.679940 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20240506 15:40:23.680644 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.668693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:42311 (local address 127.17.240.253:38535)
0506 15:40:23.668757 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.668759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.668927 (+ 168us) server_negotiation.cc:407] Connection header received
0506 15:40:23.669594 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.669595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.669608 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.669638 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.670934 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.671758 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.672603 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.672813 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.676712 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.676716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.676717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.676726 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.678693 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.679107 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.679108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.679108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.679137 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.679311 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.679312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.679313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.679443 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.679533 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.680304 (+ 771us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.680409 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:40:23.680727 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.668680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:39311)
0506 15:40:23.668832 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.668846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.669451 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.669780 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.669785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.670320 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.670798 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.670804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.671858 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.671859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.672453 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.672455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.672810 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.673647 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.673651 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.675713 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.678426 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.678438 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.678441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.678782 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.678999 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.679000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.679000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.679114 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.679471 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.679477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.679647 (+ 170us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.680397 (+ 750us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.680570 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20240506 15:40:23.681133 25457 heartbeater.cc:347] Connected to a master server at 127.17.240.253:38535
I20240506 15:40:23.681155 25456 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43707
I20240506 15:40:23.681246 25456 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:23.681245 25457 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:23.681492 25456 heartbeater.cc:511] Master 127.17.240.254:43707 requested a full tablet report, sending...
I20240506 15:40:23.681492 25457 heartbeater.cc:511] Master 127.17.240.253:38535 requested a full tablet report, sending...
I20240506 15:40:23.681521 25461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.668948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:44923 (local address 127.17.240.193:33049)
0506 15:40:23.669069 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.669076 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.669473 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.669780 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.669785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.670416 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.670798 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.670804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.672130 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.672131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.672669 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.672672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.672807 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.674136 (+ 1329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.674140 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.677409 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.680373 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.680374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.680375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.680676 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.680899 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.680900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.680900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.680930 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.681214 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.681215 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.681263 (+ 48us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.681327 (+ 64us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.681427 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":84,"threads_started":1}
I20240506 15:40:23.681825 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.669196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:33049 (local address 127.17.240.252:44923)
0506 15:40:23.669303 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.669305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.669322 (+ 17us) server_negotiation.cc:407] Connection header received
0506 15:40:23.669628 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.669630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.669643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.669700 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.670918 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.672038 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.673610 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.673824 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.678408 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.678416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.678416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.678424 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.680282 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.680786 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.680788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.680788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.680827 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.680996 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.680997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.680998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.681118 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.681186 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.681582 (+ 396us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.681696 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":59,"threads_started":1}
I20240506 15:40:23.681977 24492 ts_manager.cc:194] Re-registered known tserver with Master: 9899ac0aff6c40b1bdb86a2f5d922d76 (127.17.240.193:36253)
I20240506 15:40:23.682030 25094 ts_manager.cc:194] Registered new tserver with Master: 9899ac0aff6c40b1bdb86a2f5d922d76 (127.17.240.193:36253)
I20240506 15:40:23.682128 25455 heartbeater.cc:347] Connected to a master server at 127.17.240.252:44923
I20240506 15:40:23.682176 25455 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:23.682269 25455 heartbeater.cc:511] Master 127.17.240.252:44923 requested a full tablet report, sending...
I20240506 15:40:23.682508 24492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:39311
I20240506 15:40:23.682746 25276 ts_manager.cc:194] Registered new tserver with Master: 9899ac0aff6c40b1bdb86a2f5d922d76 (127.17.240.193:36253)
W20240506 15:40:23.761767 25463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:23.762007 25463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:23.762040 25463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:23.763842 25463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:23.763898 25463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:23.763924 25463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:40:23.765458 25463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab
--principal=oryx/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:38169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=41703
--tserver_master_addrs=127.17.240.254:43707,127.17.240.253:38535,127.17.240.252:44923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:23.765663 25463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:23.765820 25463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:23.771137 25469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.771761 25471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.772387 25468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:23.773155 25463 server_base.cc:1034] running on GCE node
I20240506 15:40:23.773347 25463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:23.773568 25463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:23.774718 25463 hybrid_clock.cc:648] HybridClock initialized: now 1715010023774690 us; error 48 us; skew 500 ppm
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:23.777345 25463 init.cc:377] Logged in from keytab as oryx/127.17.240.194@KRBTEST.COM (short username oryx)
I20240506 15:40:23.777760 25463 webserver.cc:460] Webserver started at http://127.17.240.194:41703/ using document root <none> and password file <none>
I20240506 15:40:23.777923 25463 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:23.777973 25463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:23.779213 25463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:23.779726 25478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.779904 25463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:40:23.779979 25463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "780c96a5e30c48ecb3fff58900d1a0bf"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "ebdedb47eda70254c1f220008b5ae825"
server_key_iv: "bc135f6e53fb78a4c87498320dc8d8ac"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.780257 25463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:23.791817 25463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:23.793628 25463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:23.793751 25463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:23.793932 25463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:23.794229 25463 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:23.794263 25463 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.794287 25463 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:23.794313 25463 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.799685 25463 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:38169
I20240506 15:40:23.799746 25591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:38169 every 8 connection(s)
I20240506 15:40:23.800040 25463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:40:23.809468 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 25463
I20240506 15:40:23.809638 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 24822
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.252@KRBTEST.COM
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.253@KRBTEST.COM
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:23.816336 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:41639
--local_ip_for_outbound_sockets=127.17.240.195
--tserver_master_addrs=127.17.240.254:43707
--webserver_port=37273
--webserver_interface=127.17.240.195
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab
--principal=oryx/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.240.254:43707,127.17.240.253:38535
--tserver_master_addrs=127.17.240.254:43707,127.17.240.253:38535,127.17.240.252:44923 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:40:23.816584 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.800984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:39559 (local address 127.17.240.252:44923)
0506 15:40:23.801031 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.801033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.801045 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:40:23.803217 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.803219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.803234 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.803263 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.803777 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.804456 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.805181 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.805340 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.812655 (+ 7315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.812661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.812662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.812671 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.814948 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.815543 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.815545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.815546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.815572 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.815808 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.815809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.815810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.815940 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.815992 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.816362 (+ 370us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.816464 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240506 15:40:23.816922 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.800831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:38829 (local address 127.17.240.254:43707)
0506 15:40:23.800882 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.800884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.801495 (+ 611us) server_negotiation.cc:407] Connection header received
0506 15:40:23.801873 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.801875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.801890 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.801922 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.805593 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.806276 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.807773 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.807958 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.813586 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.813591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.813591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.813600 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.815874 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.816212 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.816213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.816213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.816239 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.816403 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.816403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.816404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.816505 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.816545 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.816756 (+ 211us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.816838 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240506 15:40:23.816978 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.800721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38535 (local address 127.17.240.194:38447)
0506 15:40:23.800926 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.800959 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.801675 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.801908 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.801912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.802345 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.802801 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.802806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.803779 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.803780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.804329 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.804331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.804460 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.810700 (+ 6240us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.810704 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.812873 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.815591 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.815592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.815592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.815801 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.816072 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.816073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.816074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.816107 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.816388 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.816389 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.816432 (+ 43us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.816513 (+ 81us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.816695 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20240506 15:40:23.817116 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.800719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:38447 (local address 127.17.240.253:38535)
0506 15:40:23.800849 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.800851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.801131 (+ 280us) server_negotiation.cc:407] Connection header received
0506 15:40:23.801769 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.801770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.801783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.801845 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.802914 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.803671 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.804714 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.804903 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.812986 (+ 8083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.812990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.812991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.812998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.815457 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.815877 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.815878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.815878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.815902 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.816173 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.816174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.816174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.816304 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.816362 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.816885 (+ 523us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.816981 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20240506 15:40:23.817193 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.800721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:44923 (local address 127.17.240.194:39559)
0506 15:40:23.800931 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.800958 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.801670 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.803325 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.803326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.803475 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.803680 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.803681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.804543 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.804544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.805086 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.805087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.805386 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.807035 (+ 1649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.807061 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.812496 (+ 5435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.815092 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.815102 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.815105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.815449 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.815645 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.815646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.815646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.815738 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.816022 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.816026 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.816202 (+ 176us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.816710 (+ 508us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.816820 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20240506 15:40:23.817445 25598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.800721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:38829)
0506 15:40:23.801424 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.801433 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.801774 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.801996 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.801997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.802817 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.803099 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.803101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.806370 (+ 3269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.806370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.806895 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.806897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.808009 (+ 1112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.811089 (+ 3080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.811094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.813455 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.815956 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.815957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.815958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.816127 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.816306 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.816307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.816307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.816337 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.816600 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.816601 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.816640 (+ 39us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.817007 (+ 367us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.817108 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":544,"thread_start_us":67,"threads_started":1}
I20240506 15:40:23.817935 25594 heartbeater.cc:347] Connected to a master server at 127.17.240.253:38535
I20240506 15:40:23.817935 25593 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43707
I20240506 15:40:23.817979 25592 heartbeater.cc:347] Connected to a master server at 127.17.240.252:44923
I20240506 15:40:23.818073 25593 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:23.818073 25594 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:23.818321 25593 heartbeater.cc:511] Master 127.17.240.254:43707 requested a full tablet report, sending...
I20240506 15:40:23.818323 25594 heartbeater.cc:511] Master 127.17.240.253:38535 requested a full tablet report, sending...
I20240506 15:40:23.818073 25592 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:23.818737 25592 heartbeater.cc:511] Master 127.17.240.252:44923 requested a full tablet report, sending...
I20240506 15:40:23.818774 24492 ts_manager.cc:194] Re-registered known tserver with Master: 780c96a5e30c48ecb3fff58900d1a0bf (127.17.240.194:38169)
I20240506 15:40:23.818843 25094 ts_manager.cc:194] Registered new tserver with Master: 780c96a5e30c48ecb3fff58900d1a0bf (127.17.240.194:38169)
I20240506 15:40:23.819036 25276 ts_manager.cc:194] Registered new tserver with Master: 780c96a5e30c48ecb3fff58900d1a0bf (127.17.240.194:38169)
I20240506 15:40:23.819304 24492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:38829
W20240506 15:40:23.897643 25599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:23.897857 25599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:23.897881 25599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:23.899317 25599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:23.899374 25599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:23.899401 25599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:40:23.900970 25599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab
--principal=oryx/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:41639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=37273
--tserver_master_addrs=127.17.240.254:43707,127.17.240.253:38535,127.17.240.252:44923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:23.901185 25599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:23.901396 25599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:23.906690 25607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.907213 25604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:23.907981 25605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:23.908748 25599 server_base.cc:1034] running on GCE node
I20240506 15:40:23.908959 25599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:23.909241 25599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:23.910398 25599 hybrid_clock.cc:648] HybridClock initialized: now 1715010023910375 us; error 44 us; skew 500 ppm
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:23.912971 25599 init.cc:377] Logged in from keytab as oryx/127.17.240.195@KRBTEST.COM (short username oryx)
I20240506 15:40:23.913363 25599 webserver.cc:460] Webserver started at http://127.17.240.195:37273/ using document root <none> and password file <none>
I20240506 15:40:23.913530 25599 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:23.913586 25599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:23.914784 25599 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:23.915365 25614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.915547 25599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.915613 25599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "eed1081d2c3e466393ce04310c6f5579"
format_stamp: "Formatted at 2024-05-06 15:40:22 on dist-test-slave-v0k1"
server_key: "fb976d0540dcaf1f09f8090f98ece94e"
server_key_iv: "6a2b610a188e6f9b038a0fbdd7ab7547"
server_key_version: "encryptionkey@0"
I20240506 15:40:23.915923 25599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:23.930842 25599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:23.932731 25599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:23.932832 25599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:23.933040 25599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:23.933333 25599 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:23.933367 25599 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.933388 25599 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:23.933413 25599 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:23.938774 25599 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:41639
I20240506 15:40:23.938832 25727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:41639 every 8 connection(s)
I20240506 15:40:23.939168 25599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:40:23.941897 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 25599
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.253@KRBTEST.COM
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
May 06 15:40:23 dist-test-slave-v0k1 krb5kdc[24447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010023, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.252@KRBTEST.COM
I20240506 15:40:23.949710 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.940115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:34641 (local address 127.17.240.253:38535)
0506 15:40:23.940155 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.940157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.940168 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:40:23.940945 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.940946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.940957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.941012 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.941844 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.942572 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.943346 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.943530 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.945583 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.945587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.945587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.945594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.948171 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.948694 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.948696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.948696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.948724 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.948963 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.948964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.948964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.949081 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.949127 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.949498 (+ 371us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.949586 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:40:23.950213 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.939845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:45691)
0506 15:40:23.940150 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.940159 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.940631 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.940886 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.940886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.941314 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.941670 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.941674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.942378 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.942380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.942936 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.942937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.943096 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.944056 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.944061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.945850 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.948512 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.948513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.948513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.949273 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.949441 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.949442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.949442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.949472 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.949715 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.949716 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.949755 (+ 39us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.949824 (+ 69us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.949978 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20240506 15:40:23.950644 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.939898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:45691 (local address 127.17.240.254:43707)
0506 15:40:23.939988 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.939990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.940215 (+ 225us) server_negotiation.cc:407] Connection header received
0506 15:40:23.940808 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.940809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.940820 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.940871 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.941766 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.942291 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.943052 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.943242 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.946109 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.946113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.946113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.946121 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.948427 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.949350 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.949352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.949352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.949376 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.949535 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.949535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.949536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.949635 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.949692 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.950480 (+ 788us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.950553 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20240506 15:40:23.950644 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.939897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:59105 (local address 127.17.240.252:44923)
0506 15:40:23.939952 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.939954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.940341 (+ 387us) server_negotiation.cc:407] Connection header received
0506 15:40:23.940744 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.940746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.940760 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.940808 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.942486 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.943235 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.943996 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.944176 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.947789 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.947792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.947793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.947799 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.949482 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.949930 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.949931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.949931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.949958 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.950119 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.950120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.950120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.950216 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.950257 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.950459 (+ 202us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.950554 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240506 15:40:23.950997 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.939845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38535 (local address 127.17.240.195:34641)
0506 15:40:23.940008 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.940042 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.940626 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.941031 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.941032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.941329 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.941731 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.941732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.942688 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.942689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.943222 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.943224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.943327 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.943787 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.943848 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.945480 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.948265 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.948272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.948278 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.948600 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.948796 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.948796 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.948797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.948882 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.949175 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.949180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.949336 (+ 156us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.949824 (+ 488us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.950895 (+ 1071us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":933,"thread_start_us":88,"threads_started":1}
I20240506 15:40:23.951035 25729 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43707
I20240506 15:40:23.951141 25729 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:23.951231 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.939845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:44923 (local address 127.17.240.195:59105)
0506 15:40:23.940291 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.940298 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.940636 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.940875 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.940879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.941447 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:23.941717 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.941719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.943334 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.943334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.943905 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.943908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.944224 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.946220 (+ 1996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:23.946224 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:23.947694 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:23.949603 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.949604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.949605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.949868 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.950028 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:23.950029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:23.950029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:23.950056 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:23.950303 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:23.950304 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:23.950345 (+ 41us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.950857 (+ 512us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.950946 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":146,"thread_start_us":87,"threads_started":1}
I20240506 15:40:23.951362 25729 heartbeater.cc:511] Master 127.17.240.254:43707 requested a full tablet report, sending...
I20240506 15:40:23.951787 25730 heartbeater.cc:347] Connected to a master server at 127.17.240.253:38535
I20240506 15:40:23.951848 25730 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:23.951866 24492 ts_manager.cc:194] Re-registered known tserver with Master: eed1081d2c3e466393ce04310c6f5579 (127.17.240.195:41639)
I20240506 15:40:23.951944 25730 heartbeater.cc:511] Master 127.17.240.253:38535 requested a full tablet report, sending...
I20240506 15:40:23.952235 25094 ts_manager.cc:194] Registered new tserver with Master: eed1081d2c3e466393ce04310c6f5579 (127.17.240.195:41639)
I20240506 15:40:23.952338 24492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:45691
I20240506 15:40:23.952504 25728 heartbeater.cc:347] Connected to a master server at 127.17.240.252:44923
I20240506 15:40:23.952550 25728 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:23.952723 25728 heartbeater.cc:511] Master 127.17.240.252:44923 requested a full tablet report, sending...
I20240506 15:40:23.953051 25276 ts_manager.cc:194] Registered new tserver with Master: eed1081d2c3e466393ce04310c6f5579 (127.17.240.195:41639)
I20240506 15:40:23.954280 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.945660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.17.240.254:43707)
0506 15:40:23.945995 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.945997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.946008 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:40:23.946030 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.946030 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.946042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.946166 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.947166 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.948014 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.948992 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.949219 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.951265 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.951269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.951269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.951276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.953099 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.953516 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.953517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.953517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.953542 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.953707 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.953708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.953708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.953810 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.953913 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.954082 (+ 169us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.954169 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":105,"threads_started":1}
I20240506 15:40:23.956306 24492 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:40:23.957821 24492 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:40:23.958742 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.945818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.17.240.253:38535)
0506 15:40:23.945934 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.945936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.947199 (+ 1263us) server_negotiation.cc:407] Connection header received
0506 15:40:23.947221 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.947222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.947234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.947288 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.950208 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.953713 (+ 3505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.954542 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.954761 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.956394 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.956398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.956399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.956404 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.957787 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.958090 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.958091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.958091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.958114 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.958257 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.958258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.958259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.958353 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.958446 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.958542 (+ 96us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.958629 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20240506 15:40:23.961875 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.946115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.17.240.252:44923)
0506 15:40:23.950729 (+ 4614us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.950731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.950741 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:40:23.950762 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.950762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.950772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.950828 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:23.953934 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.954665 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.955358 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.955552 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.958948 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:23.958952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:23.958952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:23.958959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:23.960739 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.961028 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.961029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.961030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.961056 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:23.961241 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:23.961242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:23.961242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:23.961335 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:23.961423 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.961651 (+ 228us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.961774 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4598,"thread_start_us":62,"threads_started":1}
I20240506 15:40:23.965106 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.962188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:36253 (local address 127.0.0.1:33262)
0506 15:40:23.962234 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.962246 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.962285 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.962897 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.962898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.962901 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:23.963125 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.963127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.964173 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.964175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.964875 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.964877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.964941 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.964945 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.964995 (+ 50us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.965035 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20240506 15:40:23.965582 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.962507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.17.240.193:36253)
0506 15:40:23.962706 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.962716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.962727 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:40:23.962769 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.962771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.962790 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.963220 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:23.963303 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.964104 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.964979 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.965419 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.965461 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.965499 (+ 38us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.965523 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
I20240506 15:40:23.966074 25389 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea76c55af2a4b6db3159d9f4758fb9f (DEFAULT_TABLE table=test-table [id=bcc0e6963c2f413eb41f4a2abf8179e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:23.966253 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.962932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:41639 (local address 127.0.0.1:53126)
0506 15:40:23.963086 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.963097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.963146 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.963511 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.963512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.963514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:23.963715 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.963717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.965043 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.965044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.966065 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.966067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.966109 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.966111 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.966135 (+ 24us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.966184 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20240506 15:40:23.966264 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.962664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.17.240.194:38169)
0506 15:40:23.962808 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.962815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.962823 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:40:23.962854 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.962855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.962868 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.962939 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:23.963358 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.964364 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.965472 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.965959 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.966002 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.966140 (+ 138us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.966181 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20240506 15:40:23.966364 25389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea76c55af2a4b6db3159d9f4758fb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:23.966468 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.962260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:38169 (local address 127.0.0.1:56004)
0506 15:40:23.962424 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.962434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.962483 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.963038 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.963039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.963042 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:23.963267 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.963269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.964461 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.964461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.965370 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.965373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.966032 (+ 659us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.966035 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.966368 (+ 333us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.966413 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20240506 15:40:23.967053 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.963127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.17.240.195:41639)
0506 15:40:23.963277 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.963356 (+ 79us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.963366 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:40:23.963403 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.963405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.963419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.963499 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:23.963800 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.964950 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.966404 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.966874 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.966903 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.966929 (+ 26us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.966957 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20240506 15:40:23.967566 25662 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea76c55af2a4b6db3159d9f4758fb9f (DEFAULT_TABLE table=test-table [id=bcc0e6963c2f413eb41f4a2abf8179e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:23.967824 25662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea76c55af2a4b6db3159d9f4758fb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:23.969601 25754 tablet_bootstrap.cc:492] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: Bootstrap starting.
I20240506 15:40:23.970314 25754 tablet_bootstrap.cc:654] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:23.970438 25526 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea76c55af2a4b6db3159d9f4758fb9f (DEFAULT_TABLE table=test-table [id=bcc0e6963c2f413eb41f4a2abf8179e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:23.970660 25755 tablet_bootstrap.cc:492] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: Bootstrap starting.
I20240506 15:40:23.970727 25526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea76c55af2a4b6db3159d9f4758fb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:23.970885 25754 log.cc:824] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:23.971283 25755 tablet_bootstrap.cc:654] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:23.971608 25755 log.cc:824] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:23.971828 25754 tablet_bootstrap.cc:492] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: No bootstrap required, opened a new log
I20240506 15:40:23.971899 25754 ts_tablet_manager.cc:1367] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:40:23.972358 25755 tablet_bootstrap.cc:492] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: No bootstrap required, opened a new log
I20240506 15:40:23.972430 25755 ts_tablet_manager.cc:1367] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:40:23.973747 25758 tablet_bootstrap.cc:492] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: Bootstrap starting.
I20240506 15:40:23.973806 25754 raft_consensus.cc:348] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 } } peers { permanent_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38169 } } peers { permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 } }
I20240506 15:40:23.973950 25754 raft_consensus.cc:374] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:23.973977 25754 raft_consensus.cc:729] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899ac0aff6c40b1bdb86a2f5d922d76, State: Initialized, Role: FOLLOWER
I20240506 15:40:23.974098 25754 consensus_queue.cc:260] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 } } peers { permanent_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38169 } } peers { permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 } }
I20240506 15:40:23.974328 25754 ts_tablet_manager.cc:1398] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 15:40:23.974381 25758 tablet_bootstrap.cc:654] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:23.974581 25456 heartbeater.cc:503] Master 127.17.240.254:43707 was elected leader, sending a full tablet report...
I20240506 15:40:23.974673 25758 log.cc:824] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:23.974763 25755 raft_consensus.cc:348] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 } } peers { permanent_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38169 } } peers { permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 } }
I20240506 15:40:23.974900 25755 raft_consensus.cc:374] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:23.974928 25755 raft_consensus.cc:729] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed1081d2c3e466393ce04310c6f5579, State: Initialized, Role: FOLLOWER
I20240506 15:40:23.975071 25755 consensus_queue.cc:260] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 } } peers { permanent_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38169 } } peers { permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 } }
I20240506 15:40:23.975343 25755 ts_tablet_manager.cc:1398] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:40:23.975446 25729 heartbeater.cc:503] Master 127.17.240.254:43707 was elected leader, sending a full tablet report...
I20240506 15:40:23.976015 25758 tablet_bootstrap.cc:492] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: No bootstrap required, opened a new log
I20240506 15:40:23.976079 25758 ts_tablet_manager.cc:1367] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:40:23.977731 25758 raft_consensus.cc:348] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 } } peers { permanent_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38169 } } peers { permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 } }
I20240506 15:40:23.977825 25758 raft_consensus.cc:374] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:23.977841 25758 raft_consensus.cc:729] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780c96a5e30c48ecb3fff58900d1a0bf, State: Initialized, Role: FOLLOWER
I20240506 15:40:23.977911 25758 consensus_queue.cc:260] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 } } peers { permanent_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38169 } } peers { permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 } }
I20240506 15:40:23.978112 25758 ts_tablet_manager.cc:1398] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:40:23.978185 25593 heartbeater.cc:503] Master 127.17.240.254:43707 was elected leader, sending a full tablet report...
I20240506 15:40:23.988554 25762 raft_consensus.cc:482] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:40:23.988684 25762 raft_consensus.cc:504] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 } } peers { permanent_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38169 } } peers { permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 } }
I20240506 15:40:23.989025 25762 leader_election.cc:288] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9899ac0aff6c40b1bdb86a2f5d922d76 (127.17.240.193:36253), eed1081d2c3e466393ce04310c6f5579 (127.17.240.195:41639)
I20240506 15:40:23.991189 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.989211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:41639 (local address 127.17.240.194:42593)
0506 15:40:23.989323 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.989334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.989381 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.989520 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.989521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.989522 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:23.989686 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.989687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.990303 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.990304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.990906 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.990907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.990977 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.990982 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.991055 (+ 73us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.991093 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20240506 15:40:23.991194 25598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.989196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:36253 (local address 127.17.240.194:54115)
0506 15:40:23.989251 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:23.989264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:23.989310 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:23.989448 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:23.989449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:23.989451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:23.989662 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.989664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.990320 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.990320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:23.990920 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:23.990921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.990990 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.990994 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:40:23.991060 (+ 66us) client_negotiation.cc:240] Negotiation successful
0506 15:40:23.991096 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20240506 15:40:23.991657 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.989318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:42593 (local address 127.17.240.195:41639)
0506 15:40:23.989364 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.989366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.989375 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:40:23.989450 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.989451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.989459 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.989513 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:23.989762 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.990227 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.990989 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.991559 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.991566 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.991587 (+ 21us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.991611 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240506 15:40:23.991668 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:23.989264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:54115 (local address 127.17.240.193:36253)
0506 15:40:23.989314 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:23.989316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:23.989349 (+ 33us) server_negotiation.cc:407] Connection header received
0506 15:40:23.989383 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:23.989384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:23.989392 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:23.989438 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:23.989737 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.990228 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:23.991035 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:23.991539 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:23.991548 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:23.991571 (+ 23us) server_negotiation.cc:299] Negotiation successful
0506 15:40:23.991604 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:40:23.991952 25409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea76c55af2a4b6db3159d9f4758fb9f" candidate_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" is_pre_election: true
I20240506 15:40:23.991948 25682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea76c55af2a4b6db3159d9f4758fb9f" candidate_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed1081d2c3e466393ce04310c6f5579" is_pre_election: true
I20240506 15:40:23.992086 25682 raft_consensus.cc:2454] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 780c96a5e30c48ecb3fff58900d1a0bf in term 0.
I20240506 15:40:23.992094 25409 raft_consensus.cc:2454] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 780c96a5e30c48ecb3fff58900d1a0bf in term 0.
I20240506 15:40:23.992307 25482 leader_election.cc:302] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780c96a5e30c48ecb3fff58900d1a0bf, eed1081d2c3e466393ce04310c6f5579; no voters:
I20240506 15:40:23.992448 25762 raft_consensus.cc:2781] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:40:23.992511 25762 raft_consensus.cc:482] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:40:23.992551 25762 raft_consensus.cc:3037] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:23.993484 25762 raft_consensus.cc:504] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 } } peers { permanent_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38169 } } peers { permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 } }
I20240506 15:40:23.993638 25762 leader_election.cc:288] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [CANDIDATE]: Term 1 election: Requested vote from peers 9899ac0aff6c40b1bdb86a2f5d922d76 (127.17.240.193:36253), eed1081d2c3e466393ce04310c6f5579 (127.17.240.195:41639)
I20240506 15:40:23.993834 25682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea76c55af2a4b6db3159d9f4758fb9f" candidate_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed1081d2c3e466393ce04310c6f5579"
I20240506 15:40:23.993853 25409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea76c55af2a4b6db3159d9f4758fb9f" candidate_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76"
I20240506 15:40:23.993916 25409 raft_consensus.cc:3037] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:23.993916 25682 raft_consensus.cc:3037] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:23.994866 25682 raft_consensus.cc:2454] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 780c96a5e30c48ecb3fff58900d1a0bf in term 1.
I20240506 15:40:23.994978 25409 raft_consensus.cc:2454] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 780c96a5e30c48ecb3fff58900d1a0bf in term 1.
I20240506 15:40:23.995098 25482 leader_election.cc:302] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780c96a5e30c48ecb3fff58900d1a0bf, eed1081d2c3e466393ce04310c6f5579; no voters:
I20240506 15:40:23.995213 25762 raft_consensus.cc:2781] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:23.995358 25762 raft_consensus.cc:686] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 1 LEADER]: Becoming Leader. State: Replica: 780c96a5e30c48ecb3fff58900d1a0bf, State: Running, Role: LEADER
I20240506 15:40:23.995455 25762 consensus_queue.cc:237] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 } } peers { permanent_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38169 } } peers { permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 } }
I20240506 15:40:23.996294 24491 catalog_manager.cc:5484] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 780c96a5e30c48ecb3fff58900d1a0bf (127.17.240.194). New cstate: current_term: 1 leader_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "780c96a5e30c48ecb3fff58900d1a0bf" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 } health_report { overall_health: UNKNOWN } } }
I20240506 15:40:24.007231 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:24.005054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.17.240.194:38169)
0506 15:40:24.005118 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:24.005119 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:24.005127 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:40:24.005165 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:24.005166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:24.005181 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:24.005249 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:40:24.005536 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:24.006065 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:24.006654 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:24.006835 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:24.006874 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:40:24.006959 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:40:24.007053 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:24.007113 (+ 60us) server_negotiation.cc:299] Negotiation successful
0506 15:40:24.007137 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:40:24.008894 25409 raft_consensus.cc:1261] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76 [term 1 FOLLOWER]: Refusing update from remote peer 780c96a5e30c48ecb3fff58900d1a0bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:24.009192 25682 raft_consensus.cc:1261] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579 [term 1 FOLLOWER]: Refusing update from remote peer 780c96a5e30c48ecb3fff58900d1a0bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:24.009238 25763 consensus_queue.cc:1035] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9899ac0aff6c40b1bdb86a2f5d922d76" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:24.009500 25763 consensus_queue.cc:1035] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed1081d2c3e466393ce04310c6f5579" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:24.014753 25768 mvcc.cc:205] Tried to move back new op lower bound from 7024681058338070528 to 7024681058286043136. Current Snapshot: MvccSnapshot[applied={T|T < 7024681058338070528 or (T in {7024681058338070528})}]
I20240506 15:40:24.015954 24491 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37566:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:24.016065 24491 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37566:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:24.017580 24491 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0a137b87b00841f5a73fc1d4078fc43c: Sending DeleteTablet for 3 replicas of tablet 2ea76c55af2a4b6db3159d9f4758fb9f
I20240506 15:40:24.017885 25526 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea76c55af2a4b6db3159d9f4758fb9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:24 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:56004
I20240506 15:40:24.018095 25662 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea76c55af2a4b6db3159d9f4758fb9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:24 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:53126
I20240506 15:40:24.018154 25775 tablet_replica.cc:330] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: stopping tablet replica
I20240506 15:40:24.018301 25389 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea76c55af2a4b6db3159d9f4758fb9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:24 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:33262
I20240506 15:40:24.018348 25775 raft_consensus.cc:2229] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:40:24.018525 25775 raft_consensus.cc:2258] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:24.018522 25777 tablet_replica.cc:330] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: stopping tablet replica
I20240506 15:40:24.018611 25777 raft_consensus.cc:2229] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:24.018738 25777 raft_consensus.cc:2258] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:24.019225 25775 ts_tablet_manager.cc:1875] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:24.019237 25777 ts_tablet_manager.cc:1875] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:24.019896 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 24460
I20240506 15:40:24.021173 25777 ts_tablet_manager.cc:1888] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:24.021253 25777 log.cc:1196] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/2ea76c55af2a4b6db3159d9f4758fb9f
I20240506 15:40:24.021533 25777 ts_tablet_manager.cc:1909] T 2ea76c55af2a4b6db3159d9f4758fb9f P 9899ac0aff6c40b1bdb86a2f5d922d76: Deleting consensus metadata
I20240506 15:40:24.021981 25775 ts_tablet_manager.cc:1888] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:24.022115 25775 log.cc:1196] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/2ea76c55af2a4b6db3159d9f4758fb9f
I20240506 15:40:24.022440 25775 ts_tablet_manager.cc:1909] T 2ea76c55af2a4b6db3159d9f4758fb9f P 780c96a5e30c48ecb3fff58900d1a0bf: Deleting consensus metadata
I20240506 15:40:24.023392 25776 tablet_replica.cc:330] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: stopping tablet replica
I20240506 15:40:24.023545 25776 raft_consensus.cc:2229] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:24.023706 25776 raft_consensus.cc:2258] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:24.024226 25776 ts_tablet_manager.cc:1875] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:24.025810 25776 ts_tablet_manager.cc:1888] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:24.025928 25776 log.cc:1196] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/2ea76c55af2a4b6db3159d9f4758fb9f
I20240506 15:40:24.026244 25776 ts_tablet_manager.cc:1909] T 2ea76c55af2a4b6db3159d9f4758fb9f P eed1081d2c3e466393ce04310c6f5579: Deleting consensus metadata
W20240506 15:40:24.027150 25616 connection.cc:535] server connection from 127.0.0.1:53126 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 15:40:24.027186 25479 connection.cc:535] server connection from 127.0.0.1:56004 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 15:40:24.027508 25342 connection.cc:535] server connection from 127.0.0.1:33262 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 15:40:24.031893 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:24.019691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.17.240.253:38535)
0506 15:40:24.019727 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:24.019728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:24.023250 (+ 3522us) server_negotiation.cc:407] Connection header received
0506 15:40:24.023317 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:24.023319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:24.023332 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:24.023391 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:24.024428 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:24.025248 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:24.026110 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:24.026326 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:24.028092 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:24.028096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:24.028096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:24.028104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:24.030127 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:24.031137 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:24.031138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:24.031139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:24.031179 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:24.031332 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:24.031333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:24.031333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:24.031420 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:24.031521 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:24.031709 (+ 188us) server_negotiation.cc:299] Negotiation successful
0506 15:40:24.031799 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:40:24.037639 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:24.029366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.17.240.252:44923)
0506 15:40:24.029415 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:24.029417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:24.029471 (+ 54us) server_negotiation.cc:407] Connection header received
0506 15:40:24.029540 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:24.029541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:24.029554 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:24.029620 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:24.030842 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:24.031699 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:24.033147 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:24.033333 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:24.034650 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:24.034655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:24.034655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:24.034661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:24.036425 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:24.036831 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:24.036832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:24.036833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:24.036857 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:24.037010 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:24.037010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:24.037011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:24.037112 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:24.037186 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:24.037375 (+ 189us) server_negotiation.cc:299] Negotiation successful
0506 15:40:24.037522 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240506 15:40:24.038683 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:24.029189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.17.240.253:38535)
0506 15:40:24.029234 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:24.029236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:24.029633 (+ 397us) server_negotiation.cc:407] Connection header received
0506 15:40:24.029721 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:24.029722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:24.029733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:24.029762 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:24.030707 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:24.031454 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:24.032291 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:24.032482 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:24.034259 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:24.034263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:24.034263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:24.034269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:24.037549 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:24.037918 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:24.037919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:24.037920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:24.037944 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:24.038108 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:24.038108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:24.038109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:24.038216 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:24.038284 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:24.038486 (+ 202us) server_negotiation.cc:299] Negotiation successful
0506 15:40:24.038569 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:40:24.039266 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:24.019452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.17.240.252:44923)
0506 15:40:24.019498 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:24.019501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:24.032264 (+ 12763us) server_negotiation.cc:407] Connection header received
0506 15:40:24.032363 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:24.032365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:24.032375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:24.032403 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:24.033270 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:24.034051 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:24.034840 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:24.035097 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:24.036422 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:24.036426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:24.036426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:24.036433 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:24.038116 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:24.038448 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:24.038449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:24.038449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:24.038470 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:24.038633 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:24.038634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:24.038634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:24.038731 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:24.038817 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:24.039088 (+ 271us) server_negotiation.cc:299] Negotiation successful
0506 15:40:24.039175 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:40:25.013833 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:25.013607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:35927)
0506 15:40:25.013724 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:25.013778 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
W20240506 15:40:25.014027 25456 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:43707 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
I20240506 15:40:25.014619 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:25.014366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:36461)
0506 15:40:25.014514 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:25.014569 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":56,"threads_started":1}
W20240506 15:40:25.014770 25593 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:43707 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
I20240506 15:40:25.015156 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:25.014930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:59699)
0506 15:40:25.015073 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:25.015114 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
W20240506 15:40:25.015316 25729 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:43707 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
I20240506 15:40:25.517922 25789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a137b87b00841f5a73fc1d4078fc43c)
I20240506 15:40:25.518054 25789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } }
I20240506 15:40:25.518319 25789 leader_election.cc:288] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707)
I20240506 15:40:25.518702 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:25.518496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:38102)
0506 15:40:25.518602 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:25.518658 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
W20240506 15:40:25.518811 25078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
I20240506 15:40:25.519269 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:25.519143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:38114)
0506 15:40:25.519208 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:25.519236 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20240506 15:40:25.519385 25078 leader_election.cc:334] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707): Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
I20240506 15:40:25.519472 25078 leader_election.cc:302] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1167d7fcd2694c9488abfc0248fc57e1; no voters: 0a137b87b00841f5a73fc1d4078fc43c
I20240506 15:40:25.519582 25789 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:40:26.014647 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:26.014400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:41985)
0506 15:40:26.014536 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:26.014598 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":95,"threads_started":1}
I20240506 15:40:26.015307 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:26.015087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:46847)
0506 15:40:26.015213 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:26.015251 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20240506 15:40:26.015786 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:26.015579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:51385)
0506 15:40:26.015709 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:26.015744 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":51,"threads_started":1}
I20240506 15:40:27.015486 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:27.015266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:33813)
0506 15:40:27.015395 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:27.015436 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20240506 15:40:27.015995 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:27.015793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:38835)
0506 15:40:27.015916 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:27.015945 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20240506 15:40:27.016304 25797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:27.016123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:44551)
0506 15:40:27.016228 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:27.016261 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
I20240506 15:40:27.358777 25798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a137b87b00841f5a73fc1d4078fc43c)
I20240506 15:40:27.358878 25798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } }
I20240506 15:40:27.358997 25798 leader_election.cc:288] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707)
I20240506 15:40:27.359418 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:27.359197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:38350)
0506 15:40:27.359331 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:27.359375 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":47,"threads_started":1}
I20240506 15:40:27.359853 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:27.359738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:38356)
0506 15:40:27.359798 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:27.359827 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20240506 15:40:27.359968 25078 leader_election.cc:334] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707): Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
I20240506 15:40:27.360038 25078 leader_election.cc:302] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1167d7fcd2694c9488abfc0248fc57e1; no voters: 0a137b87b00841f5a73fc1d4078fc43c
I20240506 15:40:27.360124 25798 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:40:28.016155 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:28.015874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:48307)
0506 15:40:28.016021 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:28.016073 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20240506 15:40:28.016522 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:28.016317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:56489)
0506 15:40:28.016438 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:28.016470 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20240506 15:40:28.016968 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:28.016729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:53835)
0506 15:40:28.016890 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:28.016920 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
I20240506 15:40:29.017004 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:29.016748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:48415)
0506 15:40:29.016889 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:29.016942 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20240506 15:40:29.017303 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:29.017134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:36419)
0506 15:40:29.017226 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:29.017260 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":48,"threads_started":1}
I20240506 15:40:29.017494 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:29.017288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:50179)
0506 15:40:29.017420 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:29.017451 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":53,"threads_started":1}
I20240506 15:40:29.643885 25807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a137b87b00841f5a73fc1d4078fc43c)
I20240506 15:40:29.643991 25807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } }
I20240506 15:40:29.644174 25807 leader_election.cc:288] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707)
I20240506 15:40:29.644524 25808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:29.644341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:52944)
0506 15:40:29.644444 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:29.644484 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":47,"threads_started":1}
I20240506 15:40:29.644901 25808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:29.644810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:52960)
0506 15:40:29.644853 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:29.644877 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240506 15:40:29.644970 25078 leader_election.cc:334] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707): Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
I20240506 15:40:29.645123 25078 leader_election.cc:302] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1167d7fcd2694c9488abfc0248fc57e1; no voters: 0a137b87b00841f5a73fc1d4078fc43c
I20240506 15:40:29.645248 25807 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:40:30.017830 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:30.017562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:52843)
0506 15:40:30.017714 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:30.017766 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20240506 15:40:30.018110 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:30.017704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:44473)
0506 15:40:30.017995 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:30.018052 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":46,"threads_started":1}
I20240506 15:40:30.018419 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:30.017896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:51463)
0506 15:40:30.018346 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:30.018383 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":56,"threads_started":1}
I20240506 15:40:31.019119 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:31.018851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:40499)
0506 15:40:31.019047 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:31.019076 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":44,"threads_started":1}
I20240506 15:40:31.019089 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:31.018709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:57517)
0506 15:40:31.018963 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:31.019000 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":65,"threads_started":1}
I20240506 15:40:31.019119 25813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:31.018674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:46799)
0506 15:40:31.018909 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:31.018949 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":70,"threads_started":1}
I20240506 15:40:32.019783 25817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:32.019548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:48217)
0506 15:40:32.019692 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:32.019736 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20240506 15:40:32.019794 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:32.019539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:40871)
0506 15:40:32.019694 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:32.019732 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20240506 15:40:32.019960 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:32.019743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:38435)
0506 15:40:32.019836 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:32.019916 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":43,"threads_started":1}
I20240506 15:40:33.020440 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:33.020181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:33903)
0506 15:40:33.020330 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:33.020373 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20240506 15:40:33.020512 25820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:33.020257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:42799)
0506 15:40:33.020441 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:33.020469 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":60,"threads_started":1}
I20240506 15:40:33.020545 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:33.020285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:58599)
0506 15:40:33.020455 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:33.020488 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":42,"threads_started":1}
I20240506 15:40:34.021226 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:34.020900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:46351)
0506 15:40:34.021103 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:34.021157 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":66,"threads_started":1}
I20240506 15:40:34.021363 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:34.021110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:42749)
0506 15:40:34.021275 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:34.021311 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":50,"threads_started":1}
I20240506 15:40:34.021423 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:34.021144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:51717)
0506 15:40:34.021346 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:34.021375 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20240506 15:40:34.194622 25825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a137b87b00841f5a73fc1d4078fc43c)
I20240506 15:40:34.194741 25825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } }
I20240506 15:40:34.194923 25825 leader_election.cc:288] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707)
I20240506 15:40:34.195326 25826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:34.195132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:53256)
0506 15:40:34.195244 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:34.195277 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
W20240506 15:40:34.195413 25078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20240506 15:40:34.195755 25826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:34.195662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:53272)
0506 15:40:34.195713 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:34.195737 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20240506 15:40:34.195829 25078 leader_election.cc:334] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707): Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
I20240506 15:40:34.195863 25078 leader_election.cc:302] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1167d7fcd2694c9488abfc0248fc57e1; no voters: 0a137b87b00841f5a73fc1d4078fc43c
I20240506 15:40:34.195926 25825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:40:35.022127 25829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:35.021886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:46413)
0506 15:40:35.022026 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:35.022067 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20240506 15:40:35.022147 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:35.021806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:54627)
0506 15:40:35.022056 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:35.022091 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":65,"threads_started":1}
I20240506 15:40:35.022286 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:35.022093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:57941)
0506 15:40:35.022216 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:35.022243 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":44,"threads_started":1}
I20240506 15:40:36.022951 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:36.022596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:44237)
0506 15:40:36.022839 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:36.022883 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20240506 15:40:36.023100 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:36.022841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:49529)
0506 15:40:36.022980 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:36.023051 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20240506 15:40:36.023201 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:36.022786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:35883)
0506 15:40:36.023117 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:36.023151 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":45,"threads_started":1}
I20240506 15:40:37.023905 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:37.023663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:39587)
0506 15:40:37.023802 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:37.023846 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20240506 15:40:37.023905 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:37.023683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:52601)
0506 15:40:37.023808 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:37.023844 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20240506 15:40:37.024072 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:37.023727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:50169)
0506 15:40:37.023979 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:37.024013 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":58,"threads_started":1}
I20240506 15:40:37.070266 25837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a137b87b00841f5a73fc1d4078fc43c)
I20240506 15:40:37.070374 25837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } }
I20240506 15:40:37.070503 25837 leader_election.cc:288] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707)
I20240506 15:40:37.070901 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:37.070678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:53470)
0506 15:40:37.070817 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:37.070860 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20240506 15:40:37.071326 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:37.071243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:53480)
0506 15:40:37.071285 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:37.071308 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240506 15:40:37.071398 25078 leader_election.cc:334] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707): Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
I20240506 15:40:37.071468 25078 leader_election.cc:302] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1167d7fcd2694c9488abfc0248fc57e1; no voters: 0a137b87b00841f5a73fc1d4078fc43c
I20240506 15:40:37.071537 25837 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:40:38.024744 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:38.024408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:44401)
0506 15:40:38.024640 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:38.024676 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":67,"threads_started":1}
I20240506 15:40:38.024777 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:38.024425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:52893)
0506 15:40:38.024692 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:38.024726 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":53,"threads_started":1}
I20240506 15:40:38.024875 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:38.024667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:33277)
0506 15:40:38.024801 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:38.024833 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20240506 15:40:39.025550 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:39.025368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:35413)
0506 15:40:39.025486 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:39.025511 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":47,"threads_started":1}
I20240506 15:40:39.025527 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:39.025164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:47195)
0506 15:40:39.025417 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:39.025462 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":82,"threads_started":1}
I20240506 15:40:39.025619 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:39.025432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:55235)
0506 15:40:39.025539 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:39.025571 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20240506 15:40:40.026291 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:40.025946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:37537)
0506 15:40:40.026118 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:40.026223 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":62,"threads_started":1}
I20240506 15:40:40.026291 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:40.025916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:42903)
0506 15:40:40.026194 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:40.026235 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":75,"threads_started":1}
I20240506 15:40:40.026403 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:40.026079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:40477)
0506 15:40:40.026325 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:40.026361 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":52,"threads_started":1}
I20240506 15:40:41.027160 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:41.026857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:34195)
0506 15:40:41.027054 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:41.027098 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":66,"threads_started":1}
I20240506 15:40:41.027165 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:41.026857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:53553)
0506 15:40:41.027095 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:41.027129 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":69,"threads_started":1}
I20240506 15:40:41.027386 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:41.026978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:43945)
0506 15:40:41.027297 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:41.027334 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":194,"threads_started":1}
I20240506 15:40:42.027990 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:42.027745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:45347)
0506 15:40:42.027862 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:42.027937 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20240506 15:40:42.028035 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:42.027875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:57855)
0506 15:40:42.027967 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:42.027992 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":47,"threads_started":1}
I20240506 15:40:42.027979 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:42.027712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:57527)
0506 15:40:42.027872 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:42.027912 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20240506 15:40:42.145711 25855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a137b87b00841f5a73fc1d4078fc43c)
I20240506 15:40:42.145820 25855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a137b87b00841f5a73fc1d4078fc43c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43707 } } peers { permanent_uuid: "1167d7fcd2694c9488abfc0248fc57e1" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38535 } attrs { promote: false } } peers { permanent_uuid: "a512f3f825a341bb99bd83daa7178abe" member_type: NON_VOTER last_known_addr { host: "127.17.240.252" port: 44923 } attrs { promote: true } }
I20240506 15:40:42.145969 25855 leader_election.cc:288] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707)
I20240506 15:40:42.146406 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:42.146206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:46768)
0506 15:40:42.146331 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:42.146366 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
W20240506 15:40:42.146490 25078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240506 15:40:42.146888 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:42.146757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.0.0.1:46774)
0506 15:40:42.146823 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:42.146856 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20240506 15:40:42.146991 25078 leader_election.cc:334] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a137b87b00841f5a73fc1d4078fc43c (127.17.240.254:43707): Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
I20240506 15:40:42.147073 25078 leader_election.cc:302] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1167d7fcd2694c9488abfc0248fc57e1; no voters: 0a137b87b00841f5a73fc1d4078fc43c
I20240506 15:40:42.147140 25855 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1167d7fcd2694c9488abfc0248fc57e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:40:43.028692 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:43.028411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:54363)
0506 15:40:43.028582 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:43.028627 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20240506 15:40:43.028837 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:43.028559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:49305)
0506 15:40:43.028742 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:43.028780 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20240506 15:40:43.028980 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:43.028408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:53645)
0506 15:40:43.028897 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:43.028932 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":66,"threads_started":1}
I20240506 15:40:44.029613 25862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:44.029411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:58133)
0506 15:40:44.029512 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:44.029554 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
I20240506 15:40:44.029635 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:44.029244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:60141)
0506 15:40:44.029508 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:44.029554 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":68,"threads_started":1}
I20240506 15:40:44.029906 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:44.029578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:49637)
0506 15:40:44.029826 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:44.029861 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":45,"threads_started":1}
I20240506 15:40:45.030602 25866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:45.030353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:60059)
0506 15:40:45.030489 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:45.030526 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20240506 15:40:45.030741 25864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:45.030166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:46631)
0506 15:40:45.030645 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:45.030687 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":64,"threads_started":1}
I20240506 15:40:45.030822 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:45.030166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:55965)
0506 15:40:45.030748 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:45.030778 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":67,"threads_started":1}
I20240506 15:40:46.031394 25867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:46.031159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:44561)
0506 15:40:46.031299 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:46.031345 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20240506 15:40:46.031550 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:46.031376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:34561)
0506 15:40:46.031484 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:46.031512 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":40,"threads_started":1}
I20240506 15:40:46.031543 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:46.031281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:60639)
0506 15:40:46.031383 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:46.031418 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
I20240506 15:40:47.032177 25870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:47.031921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:39591)
0506 15:40:47.032068 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:47.032111 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20240506 15:40:47.032240 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:47.031950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:39609)
0506 15:40:47.032174 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:47.032203 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":43,"threads_started":1}
I20240506 15:40:47.032234 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:47.031921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:48341)
0506 15:40:47.032146 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:47.032185 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20240506 15:40:48.033075 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:48.032868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:35383)
0506 15:40:48.032975 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:48.033011 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":46,"threads_started":1}
I20240506 15:40:48.033123 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:48.032816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:35465)
0506 15:40:48.033046 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:48.033076 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":53,"threads_started":1}
I20240506 15:40:48.033155 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:48.032694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:46891)
0506 15:40:48.033058 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:48.033098 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":65,"threads_started":1}
I20240506 15:40:49.033807 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:49.033559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:58879)
0506 15:40:49.033704 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:49.033743 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20240506 15:40:49.033849 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:49.033559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:45465)
0506 15:40:49.033774 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:49.033802 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":70,"threads_started":1}
I20240506 15:40:49.033989 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:49.033788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:47931)
0506 15:40:49.033882 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:49.033916 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":43,"threads_started":1}
I20240506 15:40:50.034507 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:50.034259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:43151)
0506 15:40:50.034403 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:50.034445 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20240506 15:40:50.034802 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:50.034371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:45201)
0506 15:40:50.034705 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:50.034750 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":75,"threads_started":1}
I20240506 15:40:50.034862 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:50.034600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:47549)
0506 15:40:50.034787 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:50.034816 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":53,"threads_started":1}
I20240506 15:40:51.035513 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:51.035248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:38855)
0506 15:40:51.035377 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:51.035448 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20240506 15:40:51.035610 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:51.035292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:44415)
0506 15:40:51.035516 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:51.035552 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":58,"threads_started":1}
I20240506 15:40:51.035674 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:51.035289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:51581)
0506 15:40:51.035599 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:51.035632 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":56,"threads_started":1}
I20240506 15:40:52.036360 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:52.036121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:44639)
0506 15:40:52.036252 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:52.036303 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20240506 15:40:52.036360 25886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:52.036103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:38737)
0506 15:40:52.036260 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:52.036303 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":49,"threads_started":1}
I20240506 15:40:52.036484 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:52.036087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:41017)
0506 15:40:52.036378 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:52.036419 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":64,"threads_started":1}
I20240506 15:40:53.037209 25889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:53.036905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:52299)
0506 15:40:53.037116 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:53.037153 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20240506 15:40:53.037209 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:53.037006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:39687)
0506 15:40:53.037116 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:53.037153 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20240506 15:40:53.037228 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:53.036905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:48389)
0506 15:40:53.037135 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:53.037174 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20240506 15:40:54.037909 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:54.037661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.193:46647)
0506 15:40:54.037810 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:54.037849 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20240506 15:40:54.037967 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:54.037778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.194:57133)
0506 15:40:54.037864 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:54.037889 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":42,"threads_started":1}
I20240506 15:40:54.038098 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:54.037656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43707 (local address 127.17.240.195:54343)
0506 15:40:54.037906 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:54.038050 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":60,"threads_started":1}
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.17.240.254:43707,127.17.240.253:38535,127.17.240.252:44923, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.17.240.254:43707,127.17.240.253:38535,127.17.240.252:44923, num_attempts: 1)
I20240506 15:40:54.214489 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 25238
I20240506 15:40:54.219841 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 25463
I20240506 15:40:54.225634 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 25599
I20240506 15:40:54.230484 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 24956
I20240506 15:40:54.235303 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 25058
2024-05-06T15:40:54Z chronyd exiting
I20240506 15:40:54.251865 18371 test_util.cc:175] -----------------------------------------------
I20240506 15:40:54.251937 18371 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715009978772892-18371-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32965 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:40:54 dist-test-slave-v0k1 krb5kdc[25896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:40:54 dist-test-slave-v0k1 krb5kdc[25896](info): set up 2 sockets
May 06 15:40:54 dist-test-slave-v0k1 krb5kdc[25896](info): commencing operation
krb5kdc: starting...
W20240506 15:40:55.199363 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:40:55 dist-test-slave-v0k1 krb5kdc[25896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:40:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.254.keytab.
Entry for principal oryx/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.254.keytab.
I20240506 15:40:55.322856 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:42159
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.254.keytab
--principal=oryx/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:42159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:55.403518 25909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:55.403730 25909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:55.403754 25909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:55.405073 25909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:55.405121 25909 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240506 15:40:55.405134 25909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:55.405148 25909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:55.405158 25909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:55.405170 25909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:40:55.406834 25909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:42159
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.254.keytab
--principal=oryx/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:42159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:55.407088 25909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:55.407260 25909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:55.412541 25915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:55.413234 25917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:55.414000 25914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:55.414634 25909 server_base.cc:1034] running on GCE node
I20240506 15:40:55.414907 25909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:55.415385 25909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:55.416545 25909 hybrid_clock.cc:648] HybridClock initialized: now 1715010055416536 us; error 32 us; skew 500 ppm
May 06 15:40:55 dist-test-slave-v0k1 krb5kdc[25896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010055, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:55.425318 25909 init.cc:377] Logged in from keytab as oryx/127.17.240.254@KRBTEST.COM (short username oryx)
I20240506 15:40:55.425823 25909 webserver.cc:460] Webserver started at http://127.17.240.254:44123/ using document root <none> and password file <none>
I20240506 15:40:55.426030 25909 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:55.426079 25909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:55.426198 25909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:55.427433 25909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "b41c5ec860724848a3819102eebb8007"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "4c5c13add21ca22b589c4a2eff3ada80"
server_key_iv: "99ebbe39133b11eb7fd4b7801579e7de"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.427906 25909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "b41c5ec860724848a3819102eebb8007"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "4c5c13add21ca22b589c4a2eff3ada80"
server_key_iv: "99ebbe39133b11eb7fd4b7801579e7de"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.429380 25909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:40:55.430168 25924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:55.430392 25909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240506 15:40:55.430481 25909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "b41c5ec860724848a3819102eebb8007"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "4c5c13add21ca22b589c4a2eff3ada80"
server_key_iv: "99ebbe39133b11eb7fd4b7801579e7de"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.430681 25909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:55.444604 25909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:55.446883 25909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:55.447057 25909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:55.451483 25909 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:42159
I20240506 15:40:55.451717 25986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:42159 every 8 connection(s)
I20240506 15:40:55.451870 25909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:40:55.452540 25987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:55.454787 25987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007: Bootstrap starting.
I20240506 15:40:55.455394 25987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:55.455652 25987 log.cc:824] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:55.456403 25987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007: No bootstrap required, opened a new log
I20240506 15:40:55.457646 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 25909
I20240506 15:40:55.457746 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:40:55.457916 18371 external_mini_cluster.cc:1428] Setting key 66763987f836880172b66004d510f0aa
I20240506 15:40:55.457899 25987 raft_consensus.cc:348] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c5ec860724848a3819102eebb8007" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42159 } }
I20240506 15:40:55.458039 25987 raft_consensus.cc:374] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:55.458061 25987 raft_consensus.cc:729] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41c5ec860724848a3819102eebb8007, State: Initialized, Role: FOLLOWER
I20240506 15:40:55.458237 25987 consensus_queue.cc:260] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c5ec860724848a3819102eebb8007" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42159 } }
I20240506 15:40:55.458352 25987 raft_consensus.cc:388] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:40:55.458411 25987 raft_consensus.cc:482] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:40:55.458479 25987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:55.459419 25987 raft_consensus.cc:504] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c5ec860724848a3819102eebb8007" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42159 } }
I20240506 15:40:55.459528 25987 leader_election.cc:302] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b41c5ec860724848a3819102eebb8007; no voters:
I20240506 15:40:55.459684 25987 leader_election.cc:288] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:40:55.459743 25992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:55.459894 25992 raft_consensus.cc:686] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [term 1 LEADER]: Becoming Leader. State: Replica: b41c5ec860724848a3819102eebb8007, State: Running, Role: LEADER
I20240506 15:40:55.459926 25987 sys_catalog.cc:564] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:55.460057 25992 consensus_queue.cc:237] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c5ec860724848a3819102eebb8007" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42159 } }
I20240506 15:40:55.460367 25994 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b41c5ec860724848a3819102eebb8007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c5ec860724848a3819102eebb8007" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42159 } } }
I20240506 15:40:55.460462 25994 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:55.460649 25998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:40:55.461287 25998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:40:55.461395 25993 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b41c5ec860724848a3819102eebb8007. Latest consensus state: current_term: 1 leader_uuid: "b41c5ec860724848a3819102eebb8007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c5ec860724848a3819102eebb8007" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42159 } } }
I20240506 15:40:55.461699 25993 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:55.463150 25998 catalog_manager.cc:1347] Generated new cluster ID: f9f1a5d07fb245fda13e297cd8a41733
I20240506 15:40:55.463200 25998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:40:55 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:55.468760 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.458362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.17.240.254:42159)
0506 15:40:55.458586 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:55.458599 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:55.458622 (+ 23us) server_negotiation.cc:407] Connection header received
0506 15:40:55.458718 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:55.458722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:55.458768 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:55.458892 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:55.459681 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.460852 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:55.461970 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.462199 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:55.464752 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:55.464775 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:55.464775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:55.464795 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:55.466938 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:55.467389 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:55.467392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:55.467393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:55.467460 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:55.467614 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:55.467615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:55.467615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:55.467880 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:55.468039 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:55.468131 (+ 92us) server_negotiation.cc:299] Negotiation successful
0506 15:40:55.468269 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20240506 15:40:55.473636 25998 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:40:55.474180 25998 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:40:55.482281 25998 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007: Generated new TSK 0
I20240506 15:40:55.482486 25998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab.
Entry for principal oryx/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab.
I20240506 15:40:55.515091 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:42159
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab
--principal=oryx/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:55.603425 26013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:55.603624 26013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:55.603662 26013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:55.605018 26013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:55.605068 26013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:55.605103 26013 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240506 15:40:55.605130 26013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:40:55.606767 26013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab
--principal=oryx/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.240.254:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:55.607002 26013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:55.607234 26013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:55.613194 26021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:55.613799 26018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:55.614506 26019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:55.615238 26013 server_base.cc:1034] running on GCE node
I20240506 15:40:55.615427 26013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:55.615677 26013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:55.616825 26013 hybrid_clock.cc:648] HybridClock initialized: now 1715010055616802 us; error 45 us; skew 500 ppm
May 06 15:40:55 dist-test-slave-v0k1 krb5kdc[25896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010055, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:55.619318 26013 init.cc:377] Logged in from keytab as oryx/127.17.240.193@KRBTEST.COM (short username oryx)
I20240506 15:40:55.619763 26013 webserver.cc:460] Webserver started at http://127.17.240.193:42147/ using document root <none> and password file <none>
I20240506 15:40:55.619958 26013 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:55.620023 26013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:55.620141 26013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:55.621133 26013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "6e63391a7bd249c1a5115bb976a78303"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "2ca1ea1f25093ee412464ad231307061"
server_key_iv: "86cb18d85c217d6f3a2c3d64dfb7b4c3"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.621433 26013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "6e63391a7bd249c1a5115bb976a78303"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "2ca1ea1f25093ee412464ad231307061"
server_key_iv: "86cb18d85c217d6f3a2c3d64dfb7b4c3"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.622614 26013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:40:55.623330 26028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:55.623497 26013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:55.623564 26013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "6e63391a7bd249c1a5115bb976a78303"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "2ca1ea1f25093ee412464ad231307061"
server_key_iv: "86cb18d85c217d6f3a2c3d64dfb7b4c3"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.623634 26013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:55.639627 26013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:55.641754 26013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:55.641862 26013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:55.642405 26013 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:55.642448 26013 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:55.642472 26013 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:55.642483 26013 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 06 15:40:55 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010055, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:55.649484 26013 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:40945
I20240506 15:40:55.649552 26143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:40945 every 8 connection(s)
I20240506 15:40:55.649865 26013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:40:55.650496 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 26013
I20240506 15:40:55.650584 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:40:55.650748 18371 external_mini_cluster.cc:1428] Setting key 068bc0350f2314ce386c60f81b1a5a4b
I20240506 15:40:55.650944 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.642816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:41677 (local address 127.17.240.254:42159)
0506 15:40:55.642884 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:55.642886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:55.643091 (+ 205us) server_negotiation.cc:407] Connection header received
0506 15:40:55.643619 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:55.643620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:55.643636 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:55.643698 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:55.644430 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.644922 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:55.645502 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.645686 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:55.648039 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:55.648043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:55.648044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:55.648050 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:55.649639 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:55.650056 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:55.650057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:55.650058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:55.650092 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:55.650298 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:55.650299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:55.650299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:55.650397 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:55.650459 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:55.650733 (+ 274us) server_negotiation.cc:299] Negotiation successful
0506 15:40:55.650837 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240506 15:40:55.651048 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.642776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:42159 (local address 127.17.240.193:41677)
0506 15:40:55.642956 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:55.642984 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:55.643520 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:55.643758 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:55.643762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:55.644050 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:55.644331 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:55.644334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.645010 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:55.645011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:55.645415 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:55.645417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.645562 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:55.646136 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:55.646150 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:55.647914 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:55.649709 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:55.649713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:55.649716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:55.649972 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:55.650166 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:55.650167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:55.650167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:55.650242 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:55.650457 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:55.650462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:55.650594 (+ 132us) client_negotiation.cc:769] Sending connection context
0506 15:40:55.650707 (+ 113us) client_negotiation.cc:240] Negotiation successful
0506 15:40:55.650844 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20240506 15:40:55.651635 26144 heartbeater.cc:347] Connected to a master server at 127.17.240.254:42159
I20240506 15:40:55.651754 26144 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:55.651999 26144 heartbeater.cc:511] Master 127.17.240.254:42159 requested a full tablet report, sending...
I20240506 15:40:55.652536 25941 ts_manager.cc:194] Registered new tserver with Master: 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945)
I20240506 15:40:55.653359 25941 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:41677
WARNING: no policy specified for oryx/127.17.240.194@KRBTEST.COM; defaulting to no policy
I20240506 15:40:55.658861 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.653790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:45519 (local address 127.17.240.254:42159)
0506 15:40:55.653826 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:55.653828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:55.654243 (+ 415us) server_negotiation.cc:407] Connection header received
0506 15:40:55.654301 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:55.654302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:55.654312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:55.654368 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:55.654761 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.655258 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:55.655757 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.656252 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:55.656608 (+ 356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:55.656611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:55.656611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:55.656617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:55.657931 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:55.658255 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:55.658256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:55.658257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:55.658280 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:55.658414 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:55.658415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:55.658415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:55.658497 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:55.658555 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:55.658690 (+ 135us) server_negotiation.cc:299] Negotiation successful
0506 15:40:55.658765 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240506 15:40:55.658880 26151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.653742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:42159 (local address 127.17.240.193:45519)
0506 15:40:55.654167 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:55.654177 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:55.654241 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:55.654372 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:55.654373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:55.654514 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:55.654690 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:55.654691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.655338 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:55.655339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:55.655691 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:55.655692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.655754 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:55.656257 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:55.656263 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:55.656516 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:55.658010 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:55.658011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:55.658012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:55.658191 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:55.658335 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:55.658336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:55.658336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:55.658363 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:55.658550 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:55.658551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:55.658595 (+ 44us) client_negotiation.cc:769] Sending connection context
0506 15:40:55.658664 (+ 69us) client_negotiation.cc:240] Negotiation successful
0506 15:40:55.658766 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab.
Entry for principal oryx/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab.
I20240506 15:40:55.679283 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:42159
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab
--principal=oryx/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:55.762583 26153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:55.762781 26153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:55.762807 26153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:55.764297 26153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:55.764348 26153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:55.764369 26153 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240506 15:40:55.764392 26153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:40:55.766019 26153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab
--principal=oryx/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.240.254:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:55.766206 26153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:55.766360 26153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:55.771466 26159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:55.771925 26158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:55.772634 26161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:55.773336 26153 server_base.cc:1034] running on GCE node
I20240506 15:40:55.773514 26153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:55.773738 26153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:55.774869 26153 hybrid_clock.cc:648] HybridClock initialized: now 1715010055774851 us; error 38 us; skew 500 ppm
May 06 15:40:55 dist-test-slave-v0k1 krb5kdc[25896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010055, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:55.777341 26153 init.cc:377] Logged in from keytab as oryx/127.17.240.194@KRBTEST.COM (short username oryx)
I20240506 15:40:55.777762 26153 webserver.cc:460] Webserver started at http://127.17.240.194:34225/ using document root <none> and password file <none>
I20240506 15:40:55.777946 26153 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:55.777988 26153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:55.778084 26153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:55.779099 26153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "0f84e123ad1f431381665f7e1f0286ec"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "0e9e318f29b0a1300a4fb6e1b67f3f3f"
server_key_iv: "b4ed7f07e4a93ce934283ea5b36b6615"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.779410 26153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "0f84e123ad1f431381665f7e1f0286ec"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "0e9e318f29b0a1300a4fb6e1b67f3f3f"
server_key_iv: "b4ed7f07e4a93ce934283ea5b36b6615"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.780653 26153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:55.781345 26168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:55.781512 26153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:55.781599 26153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "0f84e123ad1f431381665f7e1f0286ec"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "0e9e318f29b0a1300a4fb6e1b67f3f3f"
server_key_iv: "b4ed7f07e4a93ce934283ea5b36b6615"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.781666 26153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:55.805629 26153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:55.807827 26153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:55.807929 26153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:55.808444 26153 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:55.808487 26153 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:55.808524 26153 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:55.808580 26153 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 06 15:40:55 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010055, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:55.815476 26153 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:33271
I20240506 15:40:55.815531 26283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:33271 every 8 connection(s)
I20240506 15:40:55.815924 26153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:40:55.818260 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.808869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:54255 (local address 127.17.240.254:42159)
0506 15:40:55.808922 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:55.808925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:55.809159 (+ 234us) server_negotiation.cc:407] Connection header received
0506 15:40:55.809911 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:55.809912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:55.809928 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:55.810018 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:55.811138 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.811990 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:55.812850 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.813070 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:55.815488 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:55.815492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:55.815493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:55.815502 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:55.817096 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:55.817470 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:55.817471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:55.817472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:55.817497 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:55.817667 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:55.817667 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:55.817668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:55.817764 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:55.817823 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:55.818072 (+ 249us) server_negotiation.cc:299] Negotiation successful
0506 15:40:55.818169 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240506 15:40:55.818370 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.808844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:42159 (local address 127.17.240.194:54255)
0506 15:40:55.809015 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:55.809049 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:55.809770 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:55.810098 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:55.810104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:55.810562 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:55.811031 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:55.811038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.812103 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:55.812105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:55.812739 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:55.812741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.812893 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:55.813555 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:55.813576 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:55.815395 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:55.817187 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:55.817192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:55.817195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:55.817401 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:55.817554 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:55.817555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:55.817555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:55.817611 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:55.817818 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:55.817822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:55.817936 (+ 114us) client_negotiation.cc:769] Sending connection context
0506 15:40:55.818039 (+ 103us) client_negotiation.cc:240] Negotiation successful
0506 15:40:55.818180 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20240506 15:40:55.818982 26284 heartbeater.cc:347] Connected to a master server at 127.17.240.254:42159
I20240506 15:40:55.819105 26284 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:55.819299 26284 heartbeater.cc:511] Master 127.17.240.254:42159 requested a full tablet report, sending...
I20240506 15:40:55.819777 25941 ts_manager.cc:194] Registered new tserver with Master: 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271)
I20240506 15:40:55.820264 25941 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:54255
I20240506 15:40:55.824280 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 26153
I20240506 15:40:55.824393 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:40:55.824601 18371 external_mini_cluster.cc:1428] Setting key 24b41ba5039a8b1a20659ccb9c551515
I20240506 15:40:55.825105 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.820101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:44569 (local address 127.17.240.254:42159)
0506 15:40:55.820143 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:55.820145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:55.820249 (+ 104us) server_negotiation.cc:407] Connection header received
0506 15:40:55.820352 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:55.820353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:55.820363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:55.820415 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:55.820951 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.821399 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:55.821873 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.821972 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:55.822690 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:55.822693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:55.822694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:55.822699 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:55.824040 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:55.824345 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:55.824346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:55.824346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:55.824375 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:55.824552 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:55.824553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:55.824553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:55.824723 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:55.824795 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:55.824938 (+ 143us) server_negotiation.cc:299] Negotiation successful
0506 15:40:55.825014 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240506 15:40:55.825122 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.820046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:42159 (local address 127.17.240.194:44569)
0506 15:40:55.820183 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:55.820192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:55.820250 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:55.820550 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:55.820551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:55.820676 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:55.820861 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:55.820862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.821465 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:55.821466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:55.821817 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:55.821818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.821870 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:55.822353 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:55.822358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:55.822603 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:55.824136 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:55.824137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:55.824138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:55.824276 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:55.824459 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:55.824460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:55.824461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:55.824495 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:55.824791 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:55.824791 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:55.824830 (+ 39us) client_negotiation.cc:769] Sending connection context
0506 15:40:55.824897 (+ 67us) client_negotiation.cc:240] Negotiation successful
0506 15:40:55.825005 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for oryx/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab.
Entry for principal oryx/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab.
I20240506 15:40:55.851097 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:42159
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab
--principal=oryx/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:55.934695 26293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:55.934886 26293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:55.934911 26293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:55.936336 26293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:55.936386 26293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:55.936406 26293 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240506 15:40:55.936425 26293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:40:55.938243 26293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab
--principal=oryx/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.240.254:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:55.938474 26293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:55.938679 26293 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:55.943743 26299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:55.944384 26298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:55.945078 26301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:55.945760 26293 server_base.cc:1034] running on GCE node
I20240506 15:40:55.945952 26293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:55.946267 26293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:55.947441 26293 hybrid_clock.cc:648] HybridClock initialized: now 1715010055947423 us; error 27 us; skew 500 ppm
May 06 15:40:55 dist-test-slave-v0k1 krb5kdc[25896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010055, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:55.949995 26293 init.cc:377] Logged in from keytab as oryx/127.17.240.195@KRBTEST.COM (short username oryx)
I20240506 15:40:55.950414 26293 webserver.cc:460] Webserver started at http://127.17.240.195:44423/ using document root <none> and password file <none>
I20240506 15:40:55.950583 26293 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:55.950626 26293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:55.950827 26293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:55.951992 26293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "ac53d540b34f495abdc83c5af30dab62"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "a3bac5a77c3ddadbe6438d3398b34848"
server_key_iv: "cf4a22072c503a8e444329d3b95ee14e"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.952296 26293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "ac53d540b34f495abdc83c5af30dab62"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "a3bac5a77c3ddadbe6438d3398b34848"
server_key_iv: "cf4a22072c503a8e444329d3b95ee14e"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.953541 26293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:40:55.954187 26308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:55.954334 26293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:55.954408 26293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "ac53d540b34f495abdc83c5af30dab62"
format_stamp: "Formatted at 2024-05-06 15:40:55 on dist-test-slave-v0k1"
server_key: "a3bac5a77c3ddadbe6438d3398b34848"
server_key_iv: "cf4a22072c503a8e444329d3b95ee14e"
server_key_version: "encryptionkey@0"
I20240506 15:40:55.954464 26293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:55.994019 26293 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:55.996234 26293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:55.996364 26293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:55.996863 26293 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:55.996905 26293 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:55.996949 26293 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:55.996974 26293 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 06 15:40:55 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010055, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:56.003413 26293 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:36945
I20240506 15:40:56.003468 26423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:36945 every 8 connection(s)
I20240506 15:40:56.003805 26293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:40:56.005950 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.997275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:51469 (local address 127.17.240.254:42159)
0506 15:40:55.997341 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:55.997343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:55.997493 (+ 150us) server_negotiation.cc:407] Connection header received
0506 15:40:55.997994 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:55.997995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:55.998009 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:55.998087 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:55.999132 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:55.999977 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.000782 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.001014 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.003323 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:56.003329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:56.003329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:56.003337 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:56.004885 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:56.005219 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:56.005220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:56.005220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:56.005243 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:56.005413 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:56.005415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:56.005415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:56.005551 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:56.005615 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.005815 (+ 200us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.005886 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240506 15:40:56.006039 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:55.997234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:42159 (local address 127.17.240.195:51469)
0506 15:40:55.997379 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:55.997407 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:55.997896 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:55.998145 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:55.998149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:55.998515 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:55.998996 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:55.999002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.000072 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.000073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.000689 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.000691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.000837 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.001485 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:56.001506 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:56.003239 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:56.004964 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:56.004968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:56.004970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:56.005155 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:56.005293 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:56.005293 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:56.005293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:56.005338 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:56.005608 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:56.005610 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:56.005711 (+ 101us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.005790 (+ 79us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.005892 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":88,"threads_started":1}
I20240506 15:40:56.006469 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 26293
I20240506 15:40:56.006552 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:40:56.006596 26424 heartbeater.cc:347] Connected to a master server at 127.17.240.254:42159
I20240506 15:40:56.006668 26424 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:56.006742 18371 external_mini_cluster.cc:1428] Setting key 8990ef8d5617f0f1cc69a719b2996262
I20240506 15:40:56.006891 26424 heartbeater.cc:511] Master 127.17.240.254:42159 requested a full tablet report, sending...
I20240506 15:40:56.007298 25940 ts_manager.cc:194] Registered new tserver with Master: ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945)
I20240506 15:40:56.007664 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:40:56.007876 25940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:51469
I20240506 15:40:56.014642 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.007701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:37691 (local address 127.17.240.254:42159)
0506 15:40:56.007737 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.007739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.009758 (+ 2019us) server_negotiation.cc:407] Connection header received
0506 15:40:56.009787 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.009788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.009801 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.009877 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:56.010288 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.010788 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.011364 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.011566 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.012289 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:56.012292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:56.012293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:56.012299 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:56.013777 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:56.014040 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:56.014041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:56.014041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:56.014063 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:56.014193 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:56.014193 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:56.014194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:56.014276 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:56.014331 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.014466 (+ 135us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.014538 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240506 15:40:56.014657 26431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.009542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:42159 (local address 127.17.240.195:37691)
0506 15:40:56.009679 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.009688 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.009745 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.009879 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.009879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.010015 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:56.010194 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.010196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.010866 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.010867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.011283 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.011285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.011347 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.011953 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:56.011957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:56.012213 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:56.013853 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:56.013854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:56.013854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:56.013985 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:56.014119 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:56.014120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:56.014120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:56.014146 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:56.014324 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:56.014324 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:56.014367 (+ 43us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.014425 (+ 58us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.014528 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
May 06 15:40:56 dist-test-slave-v0k1 krb5kdc[25896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010056, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 06 15:40:56 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:40:56 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
W20240506 15:40:56.029666 25991 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:56.024345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.17.240.254:42159)
0506 15:40:56.024408 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.024410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.024419 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:40:56.024509 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.024510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.024519 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.024570 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:56.025257 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.025732 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.026312 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.026421 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.029576 (+ 3155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59412: BlockingRecv error: recv got EOF from 127.0.0.1:59412 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
May 06 15:40:56 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010056, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:56.036880 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.030649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.17.240.254:42159)
0506 15:40:56.030695 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.030698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.030756 (+ 58us) server_negotiation.cc:407] Connection header received
0506 15:40:56.030836 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.030837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.030846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.030886 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:56.031519 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.031978 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.032498 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.032682 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.034674 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:56.034677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:56.034678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:56.034683 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:56.035978 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:56.036251 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:56.036252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:56.036253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:56.036276 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:56.036410 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:56.036411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:56.036411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:56.036498 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:56.036593 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.036690 (+ 97us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.036775 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240506 15:40:56.038748 25940 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:40:56.040311 25940 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:40:56.045241 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.042292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:40945 (local address 127.0.0.1:53314)
0506 15:40:56.042406 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.042432 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.042511 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.042990 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.042993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.042995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:56.043290 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.043293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.044233 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.044234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.044922 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.044924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.045005 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.045046 (+ 41us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.045120 (+ 74us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.045161 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
I20240506 15:40:56.045711 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.042386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.17.240.193:40945)
0506 15:40:56.042621 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.042627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.042635 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:40:56.042665 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.042667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.042679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.042752 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:56.043381 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.044155 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.045025 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.045517 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.045567 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.045603 (+ 36us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.045651 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":135,"threads_started":1}
I20240506 15:40:56.045804 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.042454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:33271 (local address 127.0.0.1:55576)
0506 15:40:56.042757 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.042768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.042836 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.043088 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.043089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.043091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:56.043301 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.043303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.044739 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.044740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.045598 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.045601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.045665 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.045668 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.045702 (+ 34us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.045739 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":72,"threads_started":1}
I20240506 15:40:56.046401 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.042595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.17.240.194:33271)
0506 15:40:56.042777 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.042785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.042821 (+ 36us) server_negotiation.cc:407] Connection header received
0506 15:40:56.042911 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.042914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.042931 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.043082 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:56.043385 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.044650 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.045697 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.046225 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.046265 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.046294 (+ 29us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.046330 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20240506 15:40:56.046344 26069 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfc9aa9e00948ebbe576357165dda86 (DEFAULT_TABLE table=test-table [id=38f81a1c3c904914b2324929c8abb31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.046666 26069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfc9aa9e00948ebbe576357165dda86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.046895 26218 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfc9aa9e00948ebbe576357165dda86 (DEFAULT_TABLE table=test-table [id=38f81a1c3c904914b2324929c8abb31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.047200 26218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfc9aa9e00948ebbe576357165dda86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.047231 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.042616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:36945 (local address 127.0.0.1:56144)
0506 15:40:56.042864 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.042874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.042921 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.043208 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.043209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.043210 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:56.043381 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.043382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.046127 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.046128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.046964 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.046968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.047079 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.047083 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.047115 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.047160 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":72,"threads_started":1}
I20240506 15:40:56.047778 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.042807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.17.240.195:36945)
0506 15:40:56.042943 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.042952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.042960 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:40:56.042989 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.042990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.043004 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.043116 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:56.043461 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.044648 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.047107 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.047593 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.047631 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.047664 (+ 33us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.047708 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20240506 15:40:56.048314 26358 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfc9aa9e00948ebbe576357165dda86 (DEFAULT_TABLE table=test-table [id=38f81a1c3c904914b2324929c8abb31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.048547 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfc9aa9e00948ebbe576357165dda86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.049572 26452 tablet_bootstrap.cc:492] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: Bootstrap starting.
I20240506 15:40:56.050493 26452 tablet_bootstrap.cc:654] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.050783 26452 log.cc:824] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:56.051136 26453 tablet_bootstrap.cc:492] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: Bootstrap starting.
I20240506 15:40:56.051370 26455 tablet_bootstrap.cc:492] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: Bootstrap starting.
I20240506 15:40:56.051578 26452 tablet_bootstrap.cc:492] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: No bootstrap required, opened a new log
I20240506 15:40:56.051642 26452 ts_tablet_manager.cc:1367] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:40:56.052192 26455 tablet_bootstrap.cc:654] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.052443 26455 log.cc:824] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:56.053079 26455 tablet_bootstrap.cc:492] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: No bootstrap required, opened a new log
I20240506 15:40:56.053011 26452 raft_consensus.cc:348] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.053117 26452 raft_consensus.cc:374] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.053134 26455 ts_tablet_manager.cc:1367] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:40:56.053145 26452 raft_consensus.cc:729] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f84e123ad1f431381665f7e1f0286ec, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.053246 26452 consensus_queue.cc:260] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.053491 26452 ts_tablet_manager.cc:1398] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:40:56.053529 26284 heartbeater.cc:503] Master 127.17.240.254:42159 was elected leader, sending a full tablet report...
I20240506 15:40:56.054335 26453 tablet_bootstrap.cc:654] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.054654 26453 log.cc:824] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:56.054870 26455 raft_consensus.cc:348] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.054983 26455 raft_consensus.cc:374] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.055032 26455 raft_consensus.cc:729] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac53d540b34f495abdc83c5af30dab62, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.055140 26455 consensus_queue.cc:260] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.055361 26453 tablet_bootstrap.cc:492] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: No bootstrap required, opened a new log
I20240506 15:40:56.055384 26455 ts_tablet_manager.cc:1398] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:40:56.055428 26424 heartbeater.cc:503] Master 127.17.240.254:42159 was elected leader, sending a full tablet report...
I20240506 15:40:56.055464 26453 ts_tablet_manager.cc:1367] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240506 15:40:56.056970 26453 raft_consensus.cc:348] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.057097 26453 raft_consensus.cc:374] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.057127 26453 raft_consensus.cc:729] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e63391a7bd249c1a5115bb976a78303, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.057216 26453 consensus_queue.cc:260] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.057449 26453 ts_tablet_manager.cc:1398] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:40:56.057479 26144 heartbeater.cc:503] Master 127.17.240.254:42159 was elected leader, sending a full tablet report...
W20240506 15:40:56.066536 26285 tablet.cc:2369] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:40:56.102735 26460 raft_consensus.cc:482] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:40:56.102861 26460 raft_consensus.cc:504] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.103240 26460 leader_election.cc:288] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271), ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945)
I20240506 15:40:56.105485 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.103413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:33271 (local address 127.17.240.193:48259)
0506 15:40:56.103473 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.103485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.103530 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.103697 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.103698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.103700 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:56.103926 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.103928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.104685 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.104687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.105319 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.105321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.105378 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.105382 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.105405 (+ 23us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.105431 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20240506 15:40:56.105690 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.103521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:36945 (local address 127.17.240.193:56209)
0506 15:40:56.103706 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.103716 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.103774 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.103892 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.103892 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.103894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:56.104166 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.104168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.104877 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.104878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.105462 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.105464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.105537 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.105541 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.105572 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.105606 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":118,"threads_started":1}
I20240506 15:40:56.106020 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.103492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:48259 (local address 127.17.240.194:33271)
0506 15:40:56.103561 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.103564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.103573 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:40:56.103604 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.103605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.103615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.103685 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:56.104008 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.104613 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.105393 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.105902 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.105911 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.105933 (+ 22us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.105968 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240506 15:40:56.106161 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.103612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:56209 (local address 127.17.240.195:36945)
0506 15:40:56.103671 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.103673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.103774 (+ 101us) server_negotiation.cc:407] Connection header received
0506 15:40:56.103821 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.103822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.103830 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.103884 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:56.104235 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.104778 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.105548 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.106036 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.106042 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.106061 (+ 19us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.106088 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240506 15:40:56.106369 26238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfc9aa9e00948ebbe576357165dda86" candidate_uuid: "6e63391a7bd249c1a5115bb976a78303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f84e123ad1f431381665f7e1f0286ec" is_pre_election: true
I20240506 15:40:56.106400 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfc9aa9e00948ebbe576357165dda86" candidate_uuid: "6e63391a7bd249c1a5115bb976a78303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac53d540b34f495abdc83c5af30dab62" is_pre_election: true
I20240506 15:40:56.106546 26378 raft_consensus.cc:2454] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e63391a7bd249c1a5115bb976a78303 in term 0.
I20240506 15:40:56.106547 26238 raft_consensus.cc:2454] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e63391a7bd249c1a5115bb976a78303 in term 0.
I20240506 15:40:56.106730 26030 leader_election.cc:302] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e63391a7bd249c1a5115bb976a78303, ac53d540b34f495abdc83c5af30dab62; no voters:
I20240506 15:40:56.106880 26460 raft_consensus.cc:2781] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:40:56.106940 26460 raft_consensus.cc:482] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:40:56.106964 26460 raft_consensus.cc:3037] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.107697 26460 raft_consensus.cc:504] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.107826 26460 leader_election.cc:288] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [CANDIDATE]: Term 1 election: Requested vote from peers 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271), ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945)
I20240506 15:40:56.108039 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfc9aa9e00948ebbe576357165dda86" candidate_uuid: "6e63391a7bd249c1a5115bb976a78303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac53d540b34f495abdc83c5af30dab62"
I20240506 15:40:56.108033 26238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfc9aa9e00948ebbe576357165dda86" candidate_uuid: "6e63391a7bd249c1a5115bb976a78303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f84e123ad1f431381665f7e1f0286ec"
I20240506 15:40:56.108100 26238 raft_consensus.cc:3037] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.108103 26378 raft_consensus.cc:3037] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.108776 26378 raft_consensus.cc:2454] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e63391a7bd249c1a5115bb976a78303 in term 1.
I20240506 15:40:56.108789 26238 raft_consensus.cc:2454] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e63391a7bd249c1a5115bb976a78303 in term 1.
I20240506 15:40:56.108958 26030 leader_election.cc:302] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f84e123ad1f431381665f7e1f0286ec, 6e63391a7bd249c1a5115bb976a78303; no voters:
I20240506 15:40:56.109042 26460 raft_consensus.cc:2781] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:56.109189 26460 raft_consensus.cc:686] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 1 LEADER]: Becoming Leader. State: Replica: 6e63391a7bd249c1a5115bb976a78303, State: Running, Role: LEADER
I20240506 15:40:56.109266 26460 consensus_queue.cc:237] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.110070 25940 catalog_manager.cc:5484] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "6e63391a7bd249c1a5115bb976a78303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } health_report { overall_health: UNKNOWN } } }
I20240506 15:40:56.119074 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.117178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.17.240.193:40945)
0506 15:40:56.117223 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.117225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.117233 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:40:56.117301 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.117302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.117318 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.117376 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:40:56.117623 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.118098 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.118622 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.118731 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.118764 (+ 33us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:40:56.118843 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:40:56.118897 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.118940 (+ 43us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.118966 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240506 15:40:56.120833 26378 raft_consensus.cc:1261] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Refusing update from remote peer 6e63391a7bd249c1a5115bb976a78303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:56.120833 26238 raft_consensus.cc:1261] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Refusing update from remote peer 6e63391a7bd249c1a5115bb976a78303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:56.121323 26462 consensus_queue.cc:1035] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:56.121507 26460 consensus_queue.cc:1035] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:56.123543 26465 mvcc.cc:205] Tried to move back new op lower bound from 7024681189867962368 to 7024681189824327680. Current Snapshot: MvccSnapshot[applied={T|T < 7024681189867962368}]
I20240506 15:40:56.124111 26467 mvcc.cc:205] Tried to move back new op lower bound from 7024681189867962368 to 7024681189824327680. Current Snapshot: MvccSnapshot[applied={T|T < 7024681189867962368 or (T in {7024681189867962368})}]
I20240506 15:40:56.125983 25941 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59422:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:56.126155 25941 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59422:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:56.126956 25941 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007: Sending DeleteTablet for 3 replicas of tablet 8bfc9aa9e00948ebbe576357165dda86
I20240506 15:40:56.127334 26069 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfc9aa9e00948ebbe576357165dda86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:56 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:53314
I20240506 15:40:56.127331 26218 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfc9aa9e00948ebbe576357165dda86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:56 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:55576
I20240506 15:40:56.127594 26358 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfc9aa9e00948ebbe576357165dda86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:56 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:56144
I20240506 15:40:56.127799 26476 tablet_replica.cc:330] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: stopping tablet replica
I20240506 15:40:56.127836 26475 tablet_replica.cc:330] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: stopping tablet replica
I20240506 15:40:56.127815 25941 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 15:40:56.127931 26475 raft_consensus.cc:2229] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 15:40:56.127955 25941 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:40:56.128015 26475 raft_consensus.cc:2258] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:56.128175 26476 raft_consensus.cc:2229] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:56.128275 26476 raft_consensus.cc:2258] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:56.128690 26476 ts_tablet_manager.cc:1875] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:56.129370 26069 tablet_service.cc:1467] Processing CreateTablet for tablet 4bea96627c28469081b4faa817c04acc (DEFAULT_TABLE table=test-table [id=d9e5329f8b1c44b8bedef91e47408533]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.129396 26358 tablet_service.cc:1467] Processing CreateTablet for tablet 4bea96627c28469081b4faa817c04acc (DEFAULT_TABLE table=test-table [id=d9e5329f8b1c44b8bedef91e47408533]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.129396 26218 tablet_service.cc:1467] Processing CreateTablet for tablet 4bea96627c28469081b4faa817c04acc (DEFAULT_TABLE table=test-table [id=d9e5329f8b1c44b8bedef91e47408533]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.129494 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bea96627c28469081b4faa817c04acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.129503 26069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bea96627c28469081b4faa817c04acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.129515 26218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bea96627c28469081b4faa817c04acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.130774 26453 tablet_bootstrap.cc:492] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: Bootstrap starting.
I20240506 15:40:56.130791 26455 tablet_bootstrap.cc:492] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: Bootstrap starting.
I20240506 15:40:56.131219 26453 tablet_bootstrap.cc:654] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.131326 26455 tablet_bootstrap.cc:654] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.131336 26474 tablet_replica.cc:330] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: stopping tablet replica
I20240506 15:40:56.131619 26452 tablet_bootstrap.cc:492] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec: Bootstrap starting.
I20240506 15:40:56.131675 26475 ts_tablet_manager.cc:1875] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:56.132162 26452 tablet_bootstrap.cc:654] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.132622 26453 tablet_bootstrap.cc:492] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: No bootstrap required, opened a new log
I20240506 15:40:56.132683 26453 ts_tablet_manager.cc:1367] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:40:56.132925 26455 tablet_bootstrap.cc:492] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: No bootstrap required, opened a new log
I20240506 15:40:56.132900 26453 raft_consensus.cc:348] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.132978 26453 raft_consensus.cc:374] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.132982 26455 ts_tablet_manager.cc:1367] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:40:56.133013 26453 raft_consensus.cc:729] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e63391a7bd249c1a5115bb976a78303, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.133062 26453 consensus_queue.cc:260] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.133080 26476 ts_tablet_manager.cc:1888] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:56.133189 26453 ts_tablet_manager.cc:1398] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240506 15:40:56.133199 26476 log.cc:1196] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/8bfc9aa9e00948ebbe576357165dda86
I20240506 15:40:56.133165 26455 raft_consensus.cc:348] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.133306 26455 raft_consensus.cc:374] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.133343 26455 raft_consensus.cc:729] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac53d540b34f495abdc83c5af30dab62, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.133400 26455 consensus_queue.cc:260] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.133515 26455 ts_tablet_manager.cc:1398] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:56.133586 26475 ts_tablet_manager.cc:1888] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:56.133613 26476 ts_tablet_manager.cc:1909] T 8bfc9aa9e00948ebbe576357165dda86 P ac53d540b34f495abdc83c5af30dab62: Deleting consensus metadata
I20240506 15:40:56.133656 26475 log.cc:1196] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/8bfc9aa9e00948ebbe576357165dda86
I20240506 15:40:56.133903 26452 tablet_bootstrap.cc:492] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec: No bootstrap required, opened a new log
I20240506 15:40:56.133955 26452 ts_tablet_manager.cc:1367] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:40:56.134033 26475 ts_tablet_manager.cc:1909] T 8bfc9aa9e00948ebbe576357165dda86 P 0f84e123ad1f431381665f7e1f0286ec: Deleting consensus metadata
I20240506 15:40:56.134028 25926 catalog_manager.cc:4830] TS ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945): tablet 8bfc9aa9e00948ebbe576357165dda86 (table test-table [id=38f81a1c3c904914b2324929c8abb31d]) successfully deleted
I20240506 15:40:56.134130 26452 raft_consensus.cc:348] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.134202 26452 raft_consensus.cc:374] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.134222 26452 raft_consensus.cc:729] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f84e123ad1f431381665f7e1f0286ec, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.134268 26452 consensus_queue.cc:260] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.134356 26474 raft_consensus.cc:2229] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:40:56.134358 26452 ts_tablet_manager.cc:1398] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:56.134383 25926 catalog_manager.cc:4830] TS 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271): tablet 8bfc9aa9e00948ebbe576357165dda86 (table test-table [id=38f81a1c3c904914b2324929c8abb31d]) successfully deleted
I20240506 15:40:56.134595 26474 raft_consensus.cc:2258] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:56.135093 26474 ts_tablet_manager.cc:1875] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:56.136637 26474 ts_tablet_manager.cc:1888] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:56.136705 26474 log.cc:1196] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/8bfc9aa9e00948ebbe576357165dda86
I20240506 15:40:56.137042 26474 ts_tablet_manager.cc:1909] T 8bfc9aa9e00948ebbe576357165dda86 P 6e63391a7bd249c1a5115bb976a78303: Deleting consensus metadata
I20240506 15:40:56.137441 25926 catalog_manager.cc:4830] TS 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945): tablet 8bfc9aa9e00948ebbe576357165dda86 (table test-table [id=38f81a1c3c904914b2324929c8abb31d]) successfully deleted
W20240506 15:40:56.150584 26145 tablet.cc:2369] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:40:56.254612 26425 tablet.cc:2369] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:40:56.440686 26460 raft_consensus.cc:482] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:40:56.440819 26460 raft_consensus.cc:504] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.440996 26460 leader_election.cc:288] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271), ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945)
I20240506 15:40:56.441249 26238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bea96627c28469081b4faa817c04acc" candidate_uuid: "6e63391a7bd249c1a5115bb976a78303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f84e123ad1f431381665f7e1f0286ec" is_pre_election: true
I20240506 15:40:56.441282 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bea96627c28469081b4faa817c04acc" candidate_uuid: "6e63391a7bd249c1a5115bb976a78303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac53d540b34f495abdc83c5af30dab62" is_pre_election: true
I20240506 15:40:56.441341 26238 raft_consensus.cc:2454] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e63391a7bd249c1a5115bb976a78303 in term 0.
I20240506 15:40:56.441357 26378 raft_consensus.cc:2454] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e63391a7bd249c1a5115bb976a78303 in term 0.
I20240506 15:40:56.441501 26030 leader_election.cc:302] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e63391a7bd249c1a5115bb976a78303, ac53d540b34f495abdc83c5af30dab62; no voters:
I20240506 15:40:56.441601 26460 raft_consensus.cc:2781] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:40:56.441643 26460 raft_consensus.cc:482] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:40:56.441664 26460 raft_consensus.cc:3037] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.442287 26460 raft_consensus.cc:504] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.442399 26460 leader_election.cc:288] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [CANDIDATE]: Term 1 election: Requested vote from peers 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271), ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945)
I20240506 15:40:56.442587 26238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bea96627c28469081b4faa817c04acc" candidate_uuid: "6e63391a7bd249c1a5115bb976a78303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f84e123ad1f431381665f7e1f0286ec"
I20240506 15:40:56.442626 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bea96627c28469081b4faa817c04acc" candidate_uuid: "6e63391a7bd249c1a5115bb976a78303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac53d540b34f495abdc83c5af30dab62"
I20240506 15:40:56.442668 26238 raft_consensus.cc:3037] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.442680 26378 raft_consensus.cc:3037] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.443338 26378 raft_consensus.cc:2454] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e63391a7bd249c1a5115bb976a78303 in term 1.
I20240506 15:40:56.443362 26238 raft_consensus.cc:2454] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e63391a7bd249c1a5115bb976a78303 in term 1.
I20240506 15:40:56.443506 26030 leader_election.cc:302] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e63391a7bd249c1a5115bb976a78303, ac53d540b34f495abdc83c5af30dab62; no voters:
I20240506 15:40:56.443594 26460 raft_consensus.cc:2781] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:56.443646 26460 raft_consensus.cc:686] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 1 LEADER]: Becoming Leader. State: Replica: 6e63391a7bd249c1a5115bb976a78303, State: Running, Role: LEADER
I20240506 15:40:56.443706 26460 consensus_queue.cc:237] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.444370 25941 catalog_manager.cc:5484] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "6e63391a7bd249c1a5115bb976a78303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } health_report { overall_health: UNKNOWN } } }
May 06 15:40:56 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010055, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.254@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:56.513859 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.507725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:42159 (local address 127.0.0.1:59432)
0506 15:40:56.507856 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.507865 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.507934 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.508091 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.508092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.508453 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:56.508625 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.508627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.509253 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.509254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.509640 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.509641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.509701 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.510210 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:56.510214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:56.511629 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:56.513097 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:56.513102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:56.513104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:56.513281 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:56.513409 (+ 128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:56.513409 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:56.513409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:56.513435 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:56.513603 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:56.513607 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:56.513647 (+ 40us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.513698 (+ 51us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.513784 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20240506 15:40:56.513923 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.507785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.17.240.254:42159)
0506 15:40:56.507856 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.507858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.507926 (+ 68us) server_negotiation.cc:407] Connection header received
0506 15:40:56.507988 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.507989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.507998 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.508047 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:56.508698 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.509187 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.509700 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.509798 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.511703 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:56.511709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:56.511709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:56.511715 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:56.513029 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:56.513338 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:56.513339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:56.513339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:56.513358 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:56.513477 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:56.513478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:56.513478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:56.513559 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:56.513603 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.513717 (+ 114us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.513838 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":43}
I20240506 15:40:56.515956 25941 catalog_manager.cc:1904] Servicing CreateTable request from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:59432:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240506 15:40:56.516151 25941 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:40:56.517527 26069 tablet_service.cc:1467] Processing CreateTablet for tablet d7e11798ba604ce989b85a7f76010186 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5d5c26e73ba4c2c8ed0e669e6760430]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240506 15:40:56.517573 26358 tablet_service.cc:1467] Processing CreateTablet for tablet d7e11798ba604ce989b85a7f76010186 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5d5c26e73ba4c2c8ed0e669e6760430]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240506 15:40:56.517581 26218 tablet_service.cc:1467] Processing CreateTablet for tablet d7e11798ba604ce989b85a7f76010186 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5d5c26e73ba4c2c8ed0e669e6760430]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240506 15:40:56.517688 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7e11798ba604ce989b85a7f76010186. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.517688 26069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7e11798ba604ce989b85a7f76010186. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.517710 26218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7e11798ba604ce989b85a7f76010186. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.518731 26452 tablet_bootstrap.cc:492] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: Bootstrap starting.
I20240506 15:40:56.519125 26452 tablet_bootstrap.cc:654] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.519131 26455 tablet_bootstrap.cc:492] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: Bootstrap starting.
I20240506 15:40:56.519132 26453 tablet_bootstrap.cc:492] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: Bootstrap starting.
I20240506 15:40:56.519482 26453 tablet_bootstrap.cc:654] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.519493 26455 tablet_bootstrap.cc:654] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.519693 26452 tablet_bootstrap.cc:492] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: No bootstrap required, opened a new log
I20240506 15:40:56.519738 26452 ts_tablet_manager.cc:1367] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:56.519887 26452 raft_consensus.cc:348] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.519960 26452 raft_consensus.cc:374] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.519979 26452 raft_consensus.cc:729] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f84e123ad1f431381665f7e1f0286ec, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.520030 26452 consensus_queue.cc:260] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.520056 26453 tablet_bootstrap.cc:492] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: No bootstrap required, opened a new log
I20240506 15:40:56.520097 26453 ts_tablet_manager.cc:1367] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:56.520133 26452 ts_tablet_manager.cc:1398] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:56.520186 26457 tablet_replica.cc:439] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } } }
I20240506 15:40:56.520222 26453 raft_consensus.cc:348] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.520251 26457 tablet_replica.cc:442] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:40:56.520272 26453 raft_consensus.cc:374] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.520291 26453 raft_consensus.cc:729] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e63391a7bd249c1a5115bb976a78303, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.520337 26453 consensus_queue.cc:260] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.520421 26453 ts_tablet_manager.cc:1398] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:56.520473 26460 tablet_replica.cc:439] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } } }
I20240506 15:40:56.520596 26460 tablet_replica.cc:442] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:40:56.520967 26455 tablet_bootstrap.cc:492] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: No bootstrap required, opened a new log
I20240506 15:40:56.521008 26455 ts_tablet_manager.cc:1367] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:40:56.521118 26455 raft_consensus.cc:348] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.521169 26455 raft_consensus.cc:374] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.521183 26455 raft_consensus.cc:729] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac53d540b34f495abdc83c5af30dab62, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.521229 26455 consensus_queue.cc:260] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.521310 26455 ts_tablet_manager.cc:1398] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:56.521365 26459 tablet_replica.cc:439] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } } }
I20240506 15:40:56.521433 26459 tablet_replica.cc:442] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:40:56.585577 26459 raft_consensus.cc:482] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:40:56.585685 26459 raft_consensus.cc:504] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.586006 26459 leader_election.cc:288] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945), 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271)
I20240506 15:40:56.589166 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.586164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:40945 (local address 127.17.240.195:54227)
0506 15:40:56.586295 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.586306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.586374 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.586592 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.586593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.586595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:56.586905 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.586907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.587861 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.587862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.588873 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.588877 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.588958 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.588961 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.589001 (+ 40us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.589035 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20240506 15:40:56.589166 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.586332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:33271 (local address 127.17.240.195:52655)
0506 15:40:56.586458 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.586465 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.586503 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.586620 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.586621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.586622 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:56.586905 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.586908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.587865 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.587866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.588876 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.588877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.588954 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.588959 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.588994 (+ 35us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.589035 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":36,"threads_started":1}
I20240506 15:40:56.589556 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.586391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:52655 (local address 127.17.240.194:33271)
0506 15:40:56.586429 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.586431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.586501 (+ 70us) server_negotiation.cc:407] Connection header received
0506 15:40:56.586545 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.586546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.586554 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.586610 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:56.586993 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.587771 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.588969 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.589436 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.589444 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.589467 (+ 23us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.589495 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:40:56.589596 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.586239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:54227 (local address 127.17.240.193:40945)
0506 15:40:56.586296 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.586298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.586423 (+ 125us) server_negotiation.cc:407] Connection header received
0506 15:40:56.586458 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.586459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.586472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.586589 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:56.586993 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.587784 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.588970 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.589437 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.589444 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.589467 (+ 23us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.589526 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:40:56.589809 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7e11798ba604ce989b85a7f76010186" candidate_uuid: "ac53d540b34f495abdc83c5af30dab62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e63391a7bd249c1a5115bb976a78303" is_pre_election: true
I20240506 15:40:56.589833 26238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7e11798ba604ce989b85a7f76010186" candidate_uuid: "ac53d540b34f495abdc83c5af30dab62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f84e123ad1f431381665f7e1f0286ec" is_pre_election: true
I20240506 15:40:56.589910 26238 raft_consensus.cc:2454] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac53d540b34f495abdc83c5af30dab62 in term 0.
I20240506 15:40:56.589922 26089 raft_consensus.cc:2454] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac53d540b34f495abdc83c5af30dab62 in term 0.
I20240506 15:40:56.590097 26310 leader_election.cc:302] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e63391a7bd249c1a5115bb976a78303, ac53d540b34f495abdc83c5af30dab62; no voters:
I20240506 15:40:56.590245 26459 raft_consensus.cc:2781] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:40:56.590310 26459 raft_consensus.cc:482] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:40:56.590334 26459 raft_consensus.cc:3037] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.591037 26459 raft_consensus.cc:504] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.591162 26459 leader_election.cc:288] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [CANDIDATE]: Term 1 election: Requested vote from peers 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945), 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271)
I20240506 15:40:56.591373 26238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7e11798ba604ce989b85a7f76010186" candidate_uuid: "ac53d540b34f495abdc83c5af30dab62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f84e123ad1f431381665f7e1f0286ec"
I20240506 15:40:56.591374 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7e11798ba604ce989b85a7f76010186" candidate_uuid: "ac53d540b34f495abdc83c5af30dab62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e63391a7bd249c1a5115bb976a78303"
I20240506 15:40:56.591450 26238 raft_consensus.cc:3037] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.591452 26089 raft_consensus.cc:3037] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.591996 26238 raft_consensus.cc:2454] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac53d540b34f495abdc83c5af30dab62 in term 1.
I20240506 15:40:56.592197 26310 leader_election.cc:302] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f84e123ad1f431381665f7e1f0286ec, ac53d540b34f495abdc83c5af30dab62; no voters:
I20240506 15:40:56.592207 26089 raft_consensus.cc:2454] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac53d540b34f495abdc83c5af30dab62 in term 1.
I20240506 15:40:56.592406 26459 raft_consensus.cc:2781] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:56.592541 26459 raft_consensus.cc:686] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [term 1 LEADER]: Becoming Leader. State: Replica: ac53d540b34f495abdc83c5af30dab62, State: Running, Role: LEADER
I20240506 15:40:56.592612 26459 consensus_queue.cc:237] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.593264 25940 catalog_manager.cc:5484] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac53d540b34f495abdc83c5af30dab62 (127.17.240.195). New cstate: current_term: 1 leader_uuid: "ac53d540b34f495abdc83c5af30dab62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } health_report { overall_health: HEALTHY } } }
I20240506 15:40:56.593119 26492 tablet_replica.cc:439] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: TxnStatusTablet state changed. Reason: New leader ac53d540b34f495abdc83c5af30dab62. Latest consensus state: current_term: 1 leader_uuid: "ac53d540b34f495abdc83c5af30dab62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } } }
I20240506 15:40:56.593518 26492 tablet_replica.cc:442] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: This TxnStatusTablet replica's current role is: LEADER
I20240506 15:40:56.593796 26494 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240506 15:40:56.593899 26494 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240506 15:40:56.616353 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.613862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:36945 (local address 127.0.0.1:56154)
0506 15:40:56.613964 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.613973 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.614012 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.614204 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.614205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.614206 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:40:56.614366 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.614368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.615331 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.615333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.615749 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.615751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.615815 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.615834 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:40:56.616172 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:40:56.616183 (+ 11us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.616232 (+ 49us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.616281 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20240506 15:40:56.616365 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.613947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.17.240.195:36945)
0506 15:40:56.614062 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.614064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.614072 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:40:56.614110 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.614111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.614128 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.614193 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:40:56.614430 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.615259 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.615819 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.615976 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.616016 (+ 40us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:40:56.616109 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:40:56.616177 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.616240 (+ 63us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.616268 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20240506 15:40:56.617496 26238 raft_consensus.cc:1261] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Refusing update from remote peer ac53d540b34f495abdc83c5af30dab62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:56.617509 26089 raft_consensus.cc:1261] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Refusing update from remote peer ac53d540b34f495abdc83c5af30dab62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:56.617730 26459 consensus_queue.cc:1035] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:56.617830 26492 consensus_queue.cc:1035] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:56.618513 26460 tablet_replica.cc:439] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: TxnStatusTablet state changed. Reason: New leader ac53d540b34f495abdc83c5af30dab62. Latest consensus state: current_term: 1 leader_uuid: "ac53d540b34f495abdc83c5af30dab62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } } }
I20240506 15:40:56.618549 26457 tablet_replica.cc:439] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: TxnStatusTablet state changed. Reason: New leader ac53d540b34f495abdc83c5af30dab62. Latest consensus state: current_term: 1 leader_uuid: "ac53d540b34f495abdc83c5af30dab62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } } }
I20240506 15:40:56.618613 26460 tablet_replica.cc:442] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:40:56.618635 26457 tablet_replica.cc:442] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:40:56.619117 26492 tablet_replica.cc:439] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac53d540b34f495abdc83c5af30dab62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } } }
I20240506 15:40:56.619210 26492 tablet_replica.cc:442] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: This TxnStatusTablet replica's current role is: LEADER
I20240506 15:40:56.619278 26466 mvcc.cc:205] Tried to move back new op lower bound from 7024681191903395840 to 7024681191803854848. Current Snapshot: MvccSnapshot[applied={T|T < 7024681191903395840 or (T in {7024681191903395840})}]
I20240506 15:40:56.619524 26457 tablet_replica.cc:439] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac53d540b34f495abdc83c5af30dab62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } } }
I20240506 15:40:56.619601 26457 tablet_replica.cc:442] T d7e11798ba604ce989b85a7f76010186 P 0f84e123ad1f431381665f7e1f0286ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:40:56.620427 26459 tablet_replica.cc:439] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac53d540b34f495abdc83c5af30dab62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } } }
I20240506 15:40:56.620501 26459 tablet_replica.cc:442] T d7e11798ba604ce989b85a7f76010186 P ac53d540b34f495abdc83c5af30dab62: This TxnStatusTablet replica's current role is: LEADER
I20240506 15:40:56.620913 26460 tablet_replica.cc:439] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac53d540b34f495abdc83c5af30dab62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } } }
I20240506 15:40:56.620993 26460 tablet_replica.cc:442] T d7e11798ba604ce989b85a7f76010186 P 6e63391a7bd249c1a5115bb976a78303: This TxnStatusTablet replica's current role is: FOLLOWER
I20240506 15:40:56.627926 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.623117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:52429 (local address 127.17.240.195:36945)
0506 15:40:56.623159 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.623161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.625684 (+ 2523us) server_negotiation.cc:407] Connection header received
0506 15:40:56.625727 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.625728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.625739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.625785 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:40:56.626126 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.626673 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.627551 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.627686 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.627702 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:40:56.627723 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:40:56.627775 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.627852 (+ 77us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.627871 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:40:56.627972 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.623090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:36945 (local address 127.17.240.193:52429)
0506 15:40:56.625588 (+ 2498us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.625600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.625669 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.625796 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.625797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.625799 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:40:56.626055 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.626057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.626750 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.626751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.627465 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.627468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.627539 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.627559 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:40:56.627786 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:40:56.627788 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.627850 (+ 62us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.627892 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2448,"thread_start_us":43,"threads_started":1}
I20240506 15:40:56.630379 26378 raft_consensus.cc:1261] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Refusing update from remote peer 6e63391a7bd249c1a5115bb976a78303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:56.630417 26238 raft_consensus.cc:1261] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Refusing update from remote peer 6e63391a7bd249c1a5115bb976a78303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:56.630642 26462 consensus_queue.cc:1035] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:56.630739 26460 consensus_queue.cc:1035] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:56.641191 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.635782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:40945 (local address 127.17.240.195:49919)
0506 15:40:56.639099 (+ 3317us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.639108 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.639180 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.639293 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.639294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.639295 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:40:56.639487 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.639488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.640230 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.640231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.640689 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.640690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.640748 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.640761 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:40:56.641056 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:40:56.641057 (+ 1us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.641092 (+ 35us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.641126 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3283,"thread_start_us":53,"threads_started":1}
I20240506 15:40:56.641198 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.635821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:49919 (local address 127.17.240.193:40945)
0506 15:40:56.635858 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.635860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.639164 (+ 3304us) server_negotiation.cc:407] Connection header received
0506 15:40:56.639235 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.639236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.639247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.639291 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:40:56.639543 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.640146 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.640758 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.640941 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.640961 (+ 20us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:40:56.640993 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:40:56.641055 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.641104 (+ 49us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.641127 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240506 15:40:56.647433 25941 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59422:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:56.647557 25941 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59422:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:56.648195 25941 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007: Sending DeleteTablet for 3 replicas of tablet 4bea96627c28469081b4faa817c04acc
I20240506 15:40:56.648488 26218 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bea96627c28469081b4faa817c04acc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:56 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:55576
I20240506 15:40:56.648659 26358 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bea96627c28469081b4faa817c04acc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:56 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:56144
I20240506 15:40:56.648695 26510 tablet_replica.cc:330] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec: stopping tablet replica
I20240506 15:40:56.648759 26510 raft_consensus.cc:2229] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:56.648797 26510 raft_consensus.cc:2258] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:56.649082 26510 ts_tablet_manager.cc:1875] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:56.651173 26511 tablet_replica.cc:330] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: stopping tablet replica
I20240506 15:40:56.651234 26069 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bea96627c28469081b4faa817c04acc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:56 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:53314
I20240506 15:40:56.651284 26511 raft_consensus.cc:2229] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:56.651325 26511 raft_consensus.cc:2258] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:56.651608 26511 ts_tablet_manager.cc:1875] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:56.651633 26510 ts_tablet_manager.cc:1888] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:40:56.651710 26510 log.cc:1196] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/4bea96627c28469081b4faa817c04acc
I20240506 15:40:56.652065 26510 ts_tablet_manager.cc:1909] T 4bea96627c28469081b4faa817c04acc P 0f84e123ad1f431381665f7e1f0286ec: Deleting consensus metadata
I20240506 15:40:56.652400 25926 catalog_manager.cc:4830] TS 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271): tablet 4bea96627c28469081b4faa817c04acc (table test-table [id=d9e5329f8b1c44b8bedef91e47408533]) successfully deleted
I20240506 15:40:56.652616 26513 tablet_replica.cc:330] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: stopping tablet replica
I20240506 15:40:56.652707 26513 raft_consensus.cc:2229] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:40:56.652822 26513 raft_consensus.cc:2258] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:56.653175 26513 ts_tablet_manager.cc:1875] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:56.653297 26511 ts_tablet_manager.cc:1888] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:40:56.653414 26511 log.cc:1196] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/4bea96627c28469081b4faa817c04acc
I20240506 15:40:56.653741 26511 ts_tablet_manager.cc:1909] T 4bea96627c28469081b4faa817c04acc P ac53d540b34f495abdc83c5af30dab62: Deleting consensus metadata
I20240506 15:40:56.654063 25926 catalog_manager.cc:4830] TS ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945): tablet 4bea96627c28469081b4faa817c04acc (table test-table [id=d9e5329f8b1c44b8bedef91e47408533]) successfully deleted
I20240506 15:40:56.654771 26513 ts_tablet_manager.cc:1888] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:40:56.654834 26513 log.cc:1196] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/4bea96627c28469081b4faa817c04acc
I20240506 15:40:56.655140 26513 ts_tablet_manager.cc:1909] T 4bea96627c28469081b4faa817c04acc P 6e63391a7bd249c1a5115bb976a78303: Deleting consensus metadata
I20240506 15:40:56.655460 25926 catalog_manager.cc:4830] TS 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945): tablet 4bea96627c28469081b4faa817c04acc (table test-table [id=d9e5329f8b1c44b8bedef91e47408533]) successfully deleted
May 06 15:40:56 dist-test-slave-v0k1 krb5kdc[25896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 06 15:40:56 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:40:56 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
W20240506 15:40:56.679248 25991 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:40:56.673984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.17.240.254:42159)
0506 15:40:56.674052 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.674054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.674161 (+ 107us) server_negotiation.cc:407] Connection header received
0506 15:40:56.674247 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.674248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.674259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.674382 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:56.675111 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.675910 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.676463 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.676647 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.679179 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59444: BlockingRecv error: recv got EOF from 127.0.0.1:59444 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
May 06 15:40:56 dist-test-slave-v0k1 krb5kdc[25896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM
I20240506 15:40:56.687181 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.680137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.17.240.254:42159)
0506 15:40:56.680182 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.680183 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.680292 (+ 109us) server_negotiation.cc:407] Connection header received
0506 15:40:56.680360 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.680360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.680369 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.680414 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:56.681134 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.681606 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.682372 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.682514 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.684808 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:56.684812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:56.684812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:56.684817 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:56.686293 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:56.686620 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:56.686620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:56.686620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:56.686637 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:56.686751 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:56.686751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:56.686751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:56.686831 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:56.686901 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.687046 (+ 145us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.687115 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:40:56.688690 25941 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:40:56.688838 25941 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:40:56.690136 26069 tablet_service.cc:1467] Processing CreateTablet for tablet e54685fd70c04b8a85df561728d3d144 (DEFAULT_TABLE table=test-table [id=66acb082ce82480d9c2e48cb6e4ab8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.690141 26218 tablet_service.cc:1467] Processing CreateTablet for tablet e54685fd70c04b8a85df561728d3d144 (DEFAULT_TABLE table=test-table [id=66acb082ce82480d9c2e48cb6e4ab8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.690148 26358 tablet_service.cc:1467] Processing CreateTablet for tablet e54685fd70c04b8a85df561728d3d144 (DEFAULT_TABLE table=test-table [id=66acb082ce82480d9c2e48cb6e4ab8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.690268 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54685fd70c04b8a85df561728d3d144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.690268 26218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54685fd70c04b8a85df561728d3d144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.690269 26069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54685fd70c04b8a85df561728d3d144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.691620 26452 tablet_bootstrap.cc:492] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec: Bootstrap starting.
I20240506 15:40:56.691627 26453 tablet_bootstrap.cc:492] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303: Bootstrap starting.
I20240506 15:40:56.691978 26453 tablet_bootstrap.cc:654] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.691990 26452 tablet_bootstrap.cc:654] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.692261 26455 tablet_bootstrap.cc:492] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62: Bootstrap starting.
I20240506 15:40:56.692605 26455 tablet_bootstrap.cc:654] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.693095 26455 tablet_bootstrap.cc:492] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62: No bootstrap required, opened a new log
I20240506 15:40:56.693095 26453 tablet_bootstrap.cc:492] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303: No bootstrap required, opened a new log
I20240506 15:40:56.693149 26455 ts_tablet_manager.cc:1367] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:56.693149 26453 ts_tablet_manager.cc:1367] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:40:56.693267 26455 raft_consensus.cc:348] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.693261 26453 raft_consensus.cc:348] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.693334 26455 raft_consensus.cc:374] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.693334 26453 raft_consensus.cc:374] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.693356 26453 raft_consensus.cc:729] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e63391a7bd249c1a5115bb976a78303, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.693356 26455 raft_consensus.cc:729] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac53d540b34f495abdc83c5af30dab62, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.693403 26455 consensus_queue.cc:260] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.693401 26453 consensus_queue.cc:260] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.693485 26453 ts_tablet_manager.cc:1398] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:56.693485 26455 ts_tablet_manager.cc:1398] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:56.694049 26452 tablet_bootstrap.cc:492] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec: No bootstrap required, opened a new log
I20240506 15:40:56.694103 26452 ts_tablet_manager.cc:1367] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:40:56.694250 26452 raft_consensus.cc:348] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.694306 26452 raft_consensus.cc:374] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.694324 26452 raft_consensus.cc:729] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f84e123ad1f431381665f7e1f0286ec, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.694373 26452 consensus_queue.cc:260] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.694455 26452 ts_tablet_manager.cc:1398] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:56.733460 26457 raft_consensus.cc:482] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:40:56.733619 26457 raft_consensus.cc:504] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.733949 26457 leader_election.cc:288] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945), ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945)
I20240506 15:40:56.736595 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.734110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:40945 (local address 127.17.240.194:33695)
0506 15:40:56.734253 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.734264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.734361 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.734613 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.734614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.734616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:56.734890 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.734893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.735562 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.735562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.736360 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.736362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.736435 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.736439 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.736469 (+ 30us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.736503 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20240506 15:40:56.737015 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.734168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:33695 (local address 127.17.240.193:40945)
0506 15:40:56.734225 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.734228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.734324 (+ 96us) server_negotiation.cc:407] Connection header received
0506 15:40:56.734457 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.734459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.734469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.734516 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:56.734998 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.735485 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.736457 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.736892 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.736900 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.736924 (+ 24us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.736954 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240506 15:40:56.737195 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54685fd70c04b8a85df561728d3d144" candidate_uuid: "0f84e123ad1f431381665f7e1f0286ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e63391a7bd249c1a5115bb976a78303" is_pre_election: true
I20240506 15:40:56.737272 26089 raft_consensus.cc:2454] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f84e123ad1f431381665f7e1f0286ec in term 0.
I20240506 15:40:56.737402 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.734499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:38573 (local address 127.17.240.195:36945)
0506 15:40:56.734540 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.734542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.734550 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:40:56.734572 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.734573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.734583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.734636 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:40:56.735028 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.735739 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.736697 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.737171 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.737179 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.737290 (+ 111us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.737331 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240506 15:40:56.737666 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.734326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:36945 (local address 127.17.240.194:38573)
0506 15:40:56.734419 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:56.734432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:56.734500 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:56.734641 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:56.734641 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:56.734643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:40:56.734904 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.734906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.735815 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.735816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:56.736599 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:56.736601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.737206 (+ 605us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.737210 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:40:56.737558 (+ 348us) client_negotiation.cc:240] Negotiation successful
0506 15:40:56.737601 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":49,"threads_started":1}
I20240506 15:40:56.737751 26170 leader_election.cc:302] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f84e123ad1f431381665f7e1f0286ec, 6e63391a7bd249c1a5115bb976a78303; no voters:
I20240506 15:40:56.737890 26457 raft_consensus.cc:2781] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:40:56.737946 26457 raft_consensus.cc:482] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:40:56.737967 26457 raft_consensus.cc:3037] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.738021 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54685fd70c04b8a85df561728d3d144" candidate_uuid: "0f84e123ad1f431381665f7e1f0286ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac53d540b34f495abdc83c5af30dab62" is_pre_election: true
I20240506 15:40:56.738093 26378 raft_consensus.cc:2454] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f84e123ad1f431381665f7e1f0286ec in term 0.
I20240506 15:40:56.738680 26457 raft_consensus.cc:504] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.738785 26457 leader_election.cc:288] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [CANDIDATE]: Term 1 election: Requested vote from peers 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945), ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945)
I20240506 15:40:56.738940 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54685fd70c04b8a85df561728d3d144" candidate_uuid: "0f84e123ad1f431381665f7e1f0286ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e63391a7bd249c1a5115bb976a78303"
I20240506 15:40:56.739035 26089 raft_consensus.cc:3037] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.738981 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54685fd70c04b8a85df561728d3d144" candidate_uuid: "0f84e123ad1f431381665f7e1f0286ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac53d540b34f495abdc83c5af30dab62"
I20240506 15:40:56.739075 26378 raft_consensus.cc:3037] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:56.739838 26089 raft_consensus.cc:2454] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f84e123ad1f431381665f7e1f0286ec in term 1.
I20240506 15:40:56.739882 26378 raft_consensus.cc:2454] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f84e123ad1f431381665f7e1f0286ec in term 1.
I20240506 15:40:56.740013 26170 leader_election.cc:302] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f84e123ad1f431381665f7e1f0286ec, ac53d540b34f495abdc83c5af30dab62; no voters:
I20240506 15:40:56.740106 26457 raft_consensus.cc:2781] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:56.740245 26457 raft_consensus.cc:686] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 1 LEADER]: Becoming Leader. State: Replica: 0f84e123ad1f431381665f7e1f0286ec, State: Running, Role: LEADER
I20240506 15:40:56.740315 26457 consensus_queue.cc:237] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.740921 25940 catalog_manager.cc:5484] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194). New cstate: current_term: 1 leader_uuid: "0f84e123ad1f431381665f7e1f0286ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } health_report { overall_health: UNKNOWN } } }
I20240506 15:40:56.752957 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:56.750665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.17.240.194:33271)
0506 15:40:56.750718 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:56.750720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:56.750789 (+ 69us) server_negotiation.cc:407] Connection header received
0506 15:40:56.750868 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:56.750868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:56.750882 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:56.750939 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:40:56.751229 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.751773 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:56.752534 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:56.752651 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:56.752687 (+ 36us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:40:56.752770 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:40:56.752827 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:56.752885 (+ 58us) server_negotiation.cc:299] Negotiation successful
0506 15:40:56.752905 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240506 15:40:56.753711 26378 raft_consensus.cc:1261] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Refusing update from remote peer 0f84e123ad1f431381665f7e1f0286ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:56.753778 26089 raft_consensus.cc:1261] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Refusing update from remote peer 0f84e123ad1f431381665f7e1f0286ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:56.753947 26530 consensus_queue.cc:1035] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:56.754055 26457 consensus_queue.cc:1035] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:56.757663 25940 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59456:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:56.757746 25940 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59456:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:56.758328 25940 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007: Sending DeleteTablet for 3 replicas of tablet e54685fd70c04b8a85df561728d3d144
I20240506 15:40:56.758558 26069 tablet_service.cc:1514] Processing DeleteTablet for tablet e54685fd70c04b8a85df561728d3d144 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:56 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:53314
I20240506 15:40:56.758596 26218 tablet_service.cc:1514] Processing DeleteTablet for tablet e54685fd70c04b8a85df561728d3d144 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:56 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:55576
I20240506 15:40:56.758648 26513 tablet_replica.cc:330] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303: stopping tablet replica
I20240506 15:40:56.758678 26510 tablet_replica.cc:330] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec: stopping tablet replica
I20240506 15:40:56.758687 26513 raft_consensus.cc:2229] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:56.758718 26510 raft_consensus.cc:2229] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:40:56.758723 26513 raft_consensus.cc:2258] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:56.758778 26510 raft_consensus.cc:2258] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:56.758944 26358 tablet_service.cc:1514] Processing DeleteTablet for tablet e54685fd70c04b8a85df561728d3d144 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:56 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:56144
I20240506 15:40:56.759111 26511 tablet_replica.cc:330] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62: stopping tablet replica
I20240506 15:40:56.759160 26511 raft_consensus.cc:2229] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:56.759193 26511 raft_consensus.cc:2258] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:56.759250 26510 ts_tablet_manager.cc:1875] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:56.759788 26511 ts_tablet_manager.cc:1875] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:56.761031 26510 ts_tablet_manager.cc:1888] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:56.761080 26510 log.cc:1196] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/e54685fd70c04b8a85df561728d3d144
I20240506 15:40:56.761175 26511 ts_tablet_manager.cc:1888] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:56.761224 26511 log.cc:1196] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/e54685fd70c04b8a85df561728d3d144
I20240506 15:40:56.761319 26510 ts_tablet_manager.cc:1909] T e54685fd70c04b8a85df561728d3d144 P 0f84e123ad1f431381665f7e1f0286ec: Deleting consensus metadata
I20240506 15:40:56.761444 26511 ts_tablet_manager.cc:1909] T e54685fd70c04b8a85df561728d3d144 P ac53d540b34f495abdc83c5af30dab62: Deleting consensus metadata
I20240506 15:40:56.761607 25926 catalog_manager.cc:4830] TS 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271): tablet e54685fd70c04b8a85df561728d3d144 (table test-table [id=66acb082ce82480d9c2e48cb6e4ab8b5]) successfully deleted
I20240506 15:40:56.761765 25940 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:40:56.761914 25940 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:40:56.763468 26358 tablet_service.cc:1467] Processing CreateTablet for tablet 679ded07468f434095760d7aed7e944e (DEFAULT_TABLE table=test-table [id=33f5f5ccba524db0977f97af5d2f6734]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.763474 26069 tablet_service.cc:1467] Processing CreateTablet for tablet 679ded07468f434095760d7aed7e944e (DEFAULT_TABLE table=test-table [id=33f5f5ccba524db0977f97af5d2f6734]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.763573 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679ded07468f434095760d7aed7e944e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.763592 26069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679ded07468f434095760d7aed7e944e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.764703 26455 tablet_bootstrap.cc:492] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62: Bootstrap starting.
I20240506 15:40:56.764776 26453 tablet_bootstrap.cc:492] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303: Bootstrap starting.
I20240506 15:40:56.764981 26455 tablet_bootstrap.cc:654] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.765072 26453 tablet_bootstrap.cc:654] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.765340 26218 tablet_service.cc:1467] Processing CreateTablet for tablet 679ded07468f434095760d7aed7e944e (DEFAULT_TABLE table=test-table [id=33f5f5ccba524db0977f97af5d2f6734]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:40:56.765430 26218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679ded07468f434095760d7aed7e944e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:56.765435 26455 tablet_bootstrap.cc:492] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62: No bootstrap required, opened a new log
I20240506 15:40:56.765475 26455 ts_tablet_manager.cc:1367] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240506 15:40:56.765591 26455 raft_consensus.cc:348] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.765648 26455 raft_consensus.cc:374] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.765668 26455 raft_consensus.cc:729] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac53d540b34f495abdc83c5af30dab62, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.765708 26455 consensus_queue.cc:260] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.765785 26455 ts_tablet_manager.cc:1398] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:56.766734 26452 tablet_bootstrap.cc:492] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec: Bootstrap starting.
I20240506 15:40:56.767122 26452 tablet_bootstrap.cc:654] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:56.767571 26513 ts_tablet_manager.cc:1875] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:56.767724 26452 tablet_bootstrap.cc:492] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec: No bootstrap required, opened a new log
I20240506 15:40:56.767765 26452 ts_tablet_manager.cc:1367] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240506 15:40:56.767889 26452 raft_consensus.cc:348] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.767946 26452 raft_consensus.cc:374] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.767966 26452 raft_consensus.cc:729] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f84e123ad1f431381665f7e1f0286ec, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.768014 26452 consensus_queue.cc:260] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.768105 26452 ts_tablet_manager.cc:1398] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:56.769424 25926 catalog_manager.cc:4830] TS ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945): tablet e54685fd70c04b8a85df561728d3d144 (table test-table [id=66acb082ce82480d9c2e48cb6e4ab8b5]) successfully deleted
I20240506 15:40:56.769759 26513 ts_tablet_manager.cc:1888] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:40:56.769809 26513 log.cc:1196] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/e54685fd70c04b8a85df561728d3d144
I20240506 15:40:56.770007 26513 ts_tablet_manager.cc:1909] T e54685fd70c04b8a85df561728d3d144 P 6e63391a7bd249c1a5115bb976a78303: Deleting consensus metadata
I20240506 15:40:56.770327 25926 catalog_manager.cc:4830] TS 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945): tablet e54685fd70c04b8a85df561728d3d144 (table test-table [id=66acb082ce82480d9c2e48cb6e4ab8b5]) successfully deleted
I20240506 15:40:56.770844 26453 tablet_bootstrap.cc:492] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303: No bootstrap required, opened a new log
I20240506 15:40:56.770885 26453 ts_tablet_manager.cc:1367] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s
I20240506 15:40:56.770977 26453 raft_consensus.cc:348] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.771047 26453 raft_consensus.cc:374] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:56.771067 26453 raft_consensus.cc:729] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e63391a7bd249c1a5115bb976a78303, State: Initialized, Role: FOLLOWER
I20240506 15:40:56.771118 26453 consensus_queue.cc:260] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:56.771198 26453 ts_tablet_manager.cc:1398] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:57.123128 26457 raft_consensus.cc:482] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:40:57.123231 26457 raft_consensus.cc:504] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:57.123365 26457 leader_election.cc:288] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945), ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945)
I20240506 15:40:57.123647 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679ded07468f434095760d7aed7e944e" candidate_uuid: "0f84e123ad1f431381665f7e1f0286ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e63391a7bd249c1a5115bb976a78303" is_pre_election: true
I20240506 15:40:57.123647 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679ded07468f434095760d7aed7e944e" candidate_uuid: "0f84e123ad1f431381665f7e1f0286ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac53d540b34f495abdc83c5af30dab62" is_pre_election: true
I20240506 15:40:57.123762 26089 raft_consensus.cc:2454] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f84e123ad1f431381665f7e1f0286ec in term 0.
I20240506 15:40:57.123762 26378 raft_consensus.cc:2454] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f84e123ad1f431381665f7e1f0286ec in term 0.
I20240506 15:40:57.123919 26170 leader_election.cc:302] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f84e123ad1f431381665f7e1f0286ec, 6e63391a7bd249c1a5115bb976a78303; no voters:
I20240506 15:40:57.124034 26457 raft_consensus.cc:2781] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:40:57.124078 26457 raft_consensus.cc:482] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:40:57.124095 26457 raft_consensus.cc:3037] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:57.124742 26457 raft_consensus.cc:504] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:57.124856 26457 leader_election.cc:288] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [CANDIDATE]: Term 1 election: Requested vote from peers 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945), ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945)
I20240506 15:40:57.125048 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679ded07468f434095760d7aed7e944e" candidate_uuid: "0f84e123ad1f431381665f7e1f0286ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e63391a7bd249c1a5115bb976a78303"
I20240506 15:40:57.125063 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679ded07468f434095760d7aed7e944e" candidate_uuid: "0f84e123ad1f431381665f7e1f0286ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac53d540b34f495abdc83c5af30dab62"
I20240506 15:40:57.125123 26089 raft_consensus.cc:3037] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:57.125133 26378 raft_consensus.cc:3037] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:57.125720 26089 raft_consensus.cc:2454] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f84e123ad1f431381665f7e1f0286ec in term 1.
I20240506 15:40:57.125798 26378 raft_consensus.cc:2454] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f84e123ad1f431381665f7e1f0286ec in term 1.
I20240506 15:40:57.125890 26170 leader_election.cc:302] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f84e123ad1f431381665f7e1f0286ec, 6e63391a7bd249c1a5115bb976a78303; no voters:
I20240506 15:40:57.126003 26457 raft_consensus.cc:2781] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:40:57.126075 26457 raft_consensus.cc:686] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 1 LEADER]: Becoming Leader. State: Replica: 0f84e123ad1f431381665f7e1f0286ec, State: Running, Role: LEADER
I20240506 15:40:57.126128 26457 consensus_queue.cc:237] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } }
I20240506 15:40:57.126699 25940 catalog_manager.cc:5484] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194). New cstate: current_term: 1 leader_uuid: "0f84e123ad1f431381665f7e1f0286ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f84e123ad1f431381665f7e1f0286ec" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 } health_report { overall_health: UNKNOWN } } }
I20240506 15:40:57.215802 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:57.214000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:34101 (local address 127.17.240.195:36945)
0506 15:40:57.214048 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:57.214050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:57.214162 (+ 112us) server_negotiation.cc:407] Connection header received
0506 15:40:57.214217 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:57.214217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:57.214225 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:57.214265 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:40:57.214477 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:57.214922 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:57.215478 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:57.215596 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:57.215612 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:40:57.215636 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:40:57.215680 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:57.215731 (+ 51us) server_negotiation.cc:299] Negotiation successful
0506 15:40:57.215746 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240506 15:40:57.215816 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:57.213944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:36945 (local address 127.17.240.194:34101)
0506 15:40:57.214101 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:57.214111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:57.214165 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:57.214271 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:57.214272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:57.214273 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:40:57.214432 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:57.214433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:57.214974 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:57.214975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:57.215417 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:57.215419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:57.215476 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:57.215490 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:40:57.215687 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:40:57.215690 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:40:57.215722 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:40:57.215758 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20240506 15:40:57.217854 26378 raft_consensus.cc:1261] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Refusing update from remote peer 0f84e123ad1f431381665f7e1f0286ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:57.217862 26089 raft_consensus.cc:1261] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Refusing update from remote peer 0f84e123ad1f431381665f7e1f0286ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:40:57.218065 26535 consensus_queue.cc:1035] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac53d540b34f495abdc83c5af30dab62" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:57.218160 26457 consensus_queue.cc:1035] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e63391a7bd249c1a5115bb976a78303" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:40:57.224720 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:57.222753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:55731 (local address 127.17.240.194:33271)
0506 15:40:57.222791 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:57.222794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:57.222838 (+ 44us) server_negotiation.cc:407] Connection header received
0506 15:40:57.222912 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:57.222912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:57.222922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:57.222970 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:40:57.223326 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:57.223776 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:57.224347 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:57.224479 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:57.224495 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:40:57.224516 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:40:57.224560 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:57.224627 (+ 67us) server_negotiation.cc:299] Negotiation successful
0506 15:40:57.224647 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240506 15:40:57.224743 26549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:57.222587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:33271 (local address 127.17.240.195:55731)
0506 15:40:57.222786 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:57.222797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:57.222855 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:57.222971 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:57.222972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:57.222973 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0506 15:40:57.223173 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:57.223174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:57.223848 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:57.223849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:57.224273 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:57.224275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:57.224331 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:57.224337 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0506 15:40:57.224575 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0506 15:40:57.224576 (+ 1us) client_negotiation.cc:769] Sending connection context
0506 15:40:57.224620 (+ 44us) client_negotiation.cc:240] Negotiation successful
0506 15:40:57.224663 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
I20240506 15:40:57.230609 25940 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59456:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:57.230702 25940 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59456:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:40:57.231272 25940 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b41c5ec860724848a3819102eebb8007: Sending DeleteTablet for 3 replicas of tablet 679ded07468f434095760d7aed7e944e
I20240506 15:40:57.231487 26069 tablet_service.cc:1514] Processing DeleteTablet for tablet 679ded07468f434095760d7aed7e944e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:57 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:53314
I20240506 15:40:57.231534 26358 tablet_service.cc:1514] Processing DeleteTablet for tablet 679ded07468f434095760d7aed7e944e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:57 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:56144
I20240506 15:40:57.231586 26513 tablet_replica.cc:330] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303: stopping tablet replica
I20240506 15:40:57.231629 26511 tablet_replica.cc:330] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62: stopping tablet replica
I20240506 15:40:57.231637 26513 raft_consensus.cc:2229] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:57.231631 26218 tablet_service.cc:1514] Processing DeleteTablet for tablet 679ded07468f434095760d7aed7e944e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:40:57 UTC) from {username='oryx', principal='oryx/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:55576
I20240506 15:40:57.231689 26513 raft_consensus.cc:2258] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:57.231698 26511 raft_consensus.cc:2229] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:40:57.231711 26510 tablet_replica.cc:330] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec: stopping tablet replica
I20240506 15:40:57.231734 26511 raft_consensus.cc:2258] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:57.231765 26510 raft_consensus.cc:2229] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:40:57.231822 26510 raft_consensus.cc:2258] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:40:57.232172 26511 ts_tablet_manager.cc:1875] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:57.232198 26513 ts_tablet_manager.cc:1875] T 679ded07468f434095760d7aed7e944e P 6e63391a7bd249c1a5115bb976a78303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:57.232196 26510 ts_tablet_manager.cc:1875] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:40:57.233714 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 26013
I20240506 15:40:57.233812 26511 ts_tablet_manager.cc:1888] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:40:57.233861 26511 log.cc:1196] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/679ded07468f434095760d7aed7e944e
I20240506 15:40:57.233925 26510 ts_tablet_manager.cc:1888] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240506 15:40:57.233975 26510 log.cc:1196] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/679ded07468f434095760d7aed7e944e
I20240506 15:40:57.234051 26511 ts_tablet_manager.cc:1909] T 679ded07468f434095760d7aed7e944e P ac53d540b34f495abdc83c5af30dab62: Deleting consensus metadata
I20240506 15:40:57.234215 26510 ts_tablet_manager.cc:1909] T 679ded07468f434095760d7aed7e944e P 0f84e123ad1f431381665f7e1f0286ec: Deleting consensus metadata
I20240506 15:40:57.234392 25926 catalog_manager.cc:4830] TS ac53d540b34f495abdc83c5af30dab62 (127.17.240.195:36945): tablet 679ded07468f434095760d7aed7e944e (table test-table [id=33f5f5ccba524db0977f97af5d2f6734]) successfully deleted
I20240506 15:40:57.234457 25926 catalog_manager.cc:4830] TS 0f84e123ad1f431381665f7e1f0286ec (127.17.240.194:33271): tablet 679ded07468f434095760d7aed7e944e (table test-table [id=33f5f5ccba524db0977f97af5d2f6734]) successfully deleted
W20240506 15:40:57.240299 25926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:40945 (error 108)
I20240506 15:40:57.240511 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 26153
I20240506 15:40:57.241968 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:57.240633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:40945 (local address 127.0.0.1:53340)
0506 15:40:57.241854 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:57.241902 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:40945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1202,"thread_start_us":115,"threads_started":1}
W20240506 15:40:57.242090 25926 catalog_manager.cc:4573] TS 6e63391a7bd249c1a5115bb976a78303 (127.17.240.193:40945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 679ded07468f434095760d7aed7e944e: Network error: Client connection negotiation failed: client connection to 127.17.240.193:40945: connect: Connection refused (error 111)
I20240506 15:40:57.247064 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 26293
I20240506 15:40:57.252890 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 25909
2024-05-06T15:40:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3021 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:40:57 dist-test-slave-v0k1 krb5kdc[26555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:40:57 dist-test-slave-v0k1 krb5kdc[26555](info): set up 2 sockets
May 06 15:40:57 dist-test-slave-v0k1 krb5kdc[26555](info): commencing operation
krb5kdc: starting...
W20240506 15:40:58.211315 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:40:58 dist-test-slave-v0k1 krb5kdc[26555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:40:58.330855 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:45593
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:45593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:58.409045 26568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:58.409225 26568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:58.409246 26568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:58.410558 26568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:58.410604 26568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:58.410616 26568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:58.410630 26568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:58.410656 26568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:40:58.412329 26568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:45593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:45593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:58.412528 26568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:58.412706 26568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:58.417344 26573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:58.417855 26576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:58.418269 26574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:58.418819 26568 server_base.cc:1034] running on GCE node
I20240506 15:40:58.419054 26568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:58.419289 26568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:58.420413 26568 hybrid_clock.cc:648] HybridClock initialized: now 1715010058420401 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2024-05-06T15:40:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1173 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:40:58 dist-test-slave-v0k1 krb5kdc[26581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:40:58 dist-test-slave-v0k1 krb5kdc[26581](info): set up 2 sockets
May 06 15:40:58 dist-test-slave-v0k1 krb5kdc[26581](info): commencing operation
krb5kdc: starting...
W20240506 15:40:59.383250 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:40:59 dist-test-slave-v0k1 krb5kdc[26581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:40:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:40:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:40:59.498271 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43739
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:43739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:59.577773 26594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:59.577996 26594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:59.578030 26594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:59.579422 26594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:40:59.579468 26594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:59.579489 26594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:59.579507 26594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:40:59.579537 26594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:40:59.581135 26594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:43739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:59.581375 26594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:59.581554 26594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:59.586601 26602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:59.586959 26600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:59.587435 26599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:59.587986 26594 server_base.cc:1034] running on GCE node
I20240506 15:40:59.588189 26594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:59.588397 26594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:59.589526 26594 hybrid_clock.cc:648] HybridClock initialized: now 1715010059589514 us; error 30 us; skew 500 ppm
May 06 15:40:59 dist-test-slave-v0k1 krb5kdc[26581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010059, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:59.591967 26594 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:40:59.592404 26594 webserver.cc:460] Webserver started at http://127.17.240.254:33079/ using document root <none> and password file <none>
I20240506 15:40:59.592567 26594 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:59.592602 26594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:59.592705 26594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:59.593681 26594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "936d030ee620b69994f2a41cab02b123"
server_key_iv: "c2a429a4ad0be613ac43cf8f8ff23280"
server_key_version: "encryptionkey@0"
I20240506 15:40:59.593951 26594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "936d030ee620b69994f2a41cab02b123"
server_key_iv: "c2a429a4ad0be613ac43cf8f8ff23280"
server_key_version: "encryptionkey@0"
I20240506 15:40:59.595125 26594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:40:59.595772 26609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:59.595927 26594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:40:59.595997 26594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "936d030ee620b69994f2a41cab02b123"
server_key_iv: "c2a429a4ad0be613ac43cf8f8ff23280"
server_key_version: "encryptionkey@0"
I20240506 15:40:59.596048 26594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:59.604858 26594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:59.606849 26594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:59.606950 26594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:59.610471 26594 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:43739
I20240506 15:40:59.610523 26661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:43739 every 8 connection(s)
I20240506 15:40:59.610805 26594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:40:59.611588 26662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:40:59.612782 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 26594
I20240506 15:40:59.612882 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:40:59.613016 18371 external_mini_cluster.cc:1428] Setting key b9472924cc0a9cb3bed88e3681289b09
I20240506 15:40:59.613988 26662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8: Bootstrap starting.
I20240506 15:40:59.614637 26662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8: Neither blocks nor log segments found. Creating new log.
I20240506 15:40:59.614879 26662 log.cc:824] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8: Log is configured to *not* fsync() on all Append() calls
I20240506 15:40:59.615514 26662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8: No bootstrap required, opened a new log
I20240506 15:40:59.617214 26662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } }
I20240506 15:40:59.617331 26662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:40:59.617360 26662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab4c2274c3b4ceeaf6e06122e0818a8, State: Initialized, Role: FOLLOWER
I20240506 15:40:59.617473 26662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } }
I20240506 15:40:59.617549 26662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:40:59.617579 26662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:40:59.617609 26662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:40:59.618146 26662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } }
I20240506 15:40:59.618234 26662 leader_election.cc:302] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ab4c2274c3b4ceeaf6e06122e0818a8; no voters:
I20240506 15:40:59.618381 26662 leader_election.cc:288] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:40:59.618423 26667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 1 FOLLOWER]: Leader election won for term 1
May 06 15:40:59 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:59.618587 26667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 1 LEADER]: Becoming Leader. State: Replica: 8ab4c2274c3b4ceeaf6e06122e0818a8, State: Running, Role: LEADER
I20240506 15:40:59.618613 26662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:40:59.619230 26667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } }
I20240506 15:40:59.619613 26668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } } }
I20240506 15:40:59.619635 26669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ab4c2274c3b4ceeaf6e06122e0818a8. Latest consensus state: current_term: 1 leader_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } } }
I20240506 15:40:59.619737 26669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:59.619693 26668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [sys.catalog]: This master's current role is: LEADER
I20240506 15:40:59.619988 26676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:40:59.620499 26676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:40:59.621944 26676 catalog_manager.cc:1347] Generated new cluster ID: a8f6ed60097440dea1a5618325ccac95
I20240506 15:40:59.621989 26676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:40:59.623126 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:59.613491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.17.240.254:43739)
0506 15:40:59.613696 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:59.613708 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:59.613729 (+ 21us) server_negotiation.cc:407] Connection header received
0506 15:40:59.614262 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:59.614271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:59.614335 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:59.614498 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:59.615247 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:59.615994 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:59.616542 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:59.616783 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:59.618981 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:59.618988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:59.618989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:59.619030 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:59.621371 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:59.621779 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:59.621782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:59.621783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:59.621846 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:59.622004 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:59.622005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:59.622006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:59.622229 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:59.622731 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:59.622811 (+ 80us) server_negotiation.cc:299] Negotiation successful
0506 15:40:59.622929 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20240506 15:40:59.628501 26676 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:40:59.629024 26676 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:40:59.641304 26676 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8: Generated new TSK 0
I20240506 15:40:59.641494 26676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:40:59.675225 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43739
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:59.768203 26688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:59.768395 26688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:59.768419 26688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:59.769794 26688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:59.769846 26688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:59.769874 26688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:40:59.771580 26688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:59.771790 26688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:59.771981 26688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:59.777606 26693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:59.777921 26696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:59.778497 26688 server_base.cc:1034] running on GCE node
W20240506 15:40:59.778615 26694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:59.778844 26688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:59.779146 26688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:59.780297 26688 hybrid_clock.cc:648] HybridClock initialized: now 1715010059780283 us; error 32 us; skew 500 ppm
May 06 15:40:59 dist-test-slave-v0k1 krb5kdc[26581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010059, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:59.783071 26688 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:40:59.783516 26688 webserver.cc:460] Webserver started at http://127.17.240.193:37969/ using document root <none> and password file <none>
I20240506 15:40:59.783710 26688 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:59.783756 26688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:59.783866 26688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:59.784874 26688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "4f703877c5b841e79c40bf07016b2136"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "51e41c22172f278b5bc808b965778a6f"
server_key_iv: "c78e016081aff9652b29938c65f68a01"
server_key_version: "encryptionkey@0"
I20240506 15:40:59.785216 26688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "4f703877c5b841e79c40bf07016b2136"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "51e41c22172f278b5bc808b965778a6f"
server_key_iv: "c78e016081aff9652b29938c65f68a01"
server_key_version: "encryptionkey@0"
I20240506 15:40:59.786423 26688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:40:59.787343 26703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:59.787568 26688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:40:59.787684 26688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "4f703877c5b841e79c40bf07016b2136"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "51e41c22172f278b5bc808b965778a6f"
server_key_iv: "c78e016081aff9652b29938c65f68a01"
server_key_version: "encryptionkey@0"
I20240506 15:40:59.787750 26688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:59.802712 26688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:59.804466 26688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:59.804589 26688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:59.804787 26688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:59.805075 26688 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:59.805107 26688 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:59.805140 26688 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:59.805166 26688 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:59.810689 26688 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:37613
I20240506 15:40:59.810734 26816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:37613 every 8 connection(s)
I20240506 15:40:59.811100 26688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
May 06 15:40:59 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010059, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:59.819502 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:59.811868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:42863 (local address 127.17.240.254:43739)
0506 15:40:59.811914 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:59.811918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:59.812080 (+ 162us) server_negotiation.cc:407] Connection header received
0506 15:40:59.812571 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:59.812572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:59.812584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:59.812640 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:59.813399 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:59.813897 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:59.814426 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:59.814616 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:59.816822 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:59.816826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:59.816826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:59.816833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:59.818266 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:59.818655 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:59.818656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:59.818656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:59.818679 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:59.818844 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:59.818844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:59.818845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:59.818943 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:59.819000 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:59.819279 (+ 279us) server_negotiation.cc:299] Negotiation successful
0506 15:40:59.819385 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:40:59.819595 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:59.811820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:42863)
0506 15:40:59.811977 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:59.812006 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:59.812466 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:59.812706 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:59.812709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:59.813025 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:59.813302 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:59.813306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:59.813975 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:59.813976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:59.814350 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:59.814351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:59.814778 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:59.815337 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:59.815351 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:59.816729 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:59.818366 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:59.818370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:59.818373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:59.818586 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:59.818733 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:59.818733 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:59.818734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:59.818793 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:59.818995 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:59.819000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:59.819155 (+ 155us) client_negotiation.cc:769] Sending connection context
0506 15:40:59.819255 (+ 100us) client_negotiation.cc:240] Negotiation successful
0506 15:40:59.819398 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20240506 15:40:59.820294 26817 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43739
I20240506 15:40:59.820386 26817 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:59.820549 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 26688
I20240506 15:40:59.820632 26817 heartbeater.cc:511] Master 127.17.240.254:43739 requested a full tablet report, sending...
I20240506 15:40:59.820688 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:40:59.820887 18371 external_mini_cluster.cc:1428] Setting key 7bce36083d050da171e222934f5da045
I20240506 15:40:59.821179 26626 ts_manager.cc:194] Registered new tserver with Master: 4f703877c5b841e79c40bf07016b2136 (127.17.240.193:37613)
I20240506 15:40:59.821899 26626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:42863
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:40:59.846771 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43739
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:40:59.926219 26822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:40:59.926402 26822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:40:59.926425 26822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:40:59.927697 26822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:40:59.927742 26822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:40:59.927765 26822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:40:59.929299 26822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:40:59.929476 26822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:40:59.929630 26822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:40:59.934250 26828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:59.934685 26827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:40:59.935235 26830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:40:59.935818 26822 server_base.cc:1034] running on GCE node
I20240506 15:40:59.936009 26822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:40:59.936229 26822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:40:59.937371 26822 hybrid_clock.cc:648] HybridClock initialized: now 1715010059937357 us; error 36 us; skew 500 ppm
May 06 15:40:59 dist-test-slave-v0k1 krb5kdc[26581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010059, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:40:59.939813 26822 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:40:59.940212 26822 webserver.cc:460] Webserver started at http://127.17.240.194:43439/ using document root <none> and password file <none>
I20240506 15:40:59.940377 26822 fs_manager.cc:359] Metadata directory not provided
I20240506 15:40:59.940413 26822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:40:59.940495 26822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:40:59.941445 26822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "d4517400f4ae4e0faba63e21f081b3c2"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "5d9689b4e61a27568bdd27a54e9078e4"
server_key_iv: "954ad806559d7108e027668a94cf2e92"
server_key_version: "encryptionkey@0"
I20240506 15:40:59.941718 26822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "d4517400f4ae4e0faba63e21f081b3c2"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "5d9689b4e61a27568bdd27a54e9078e4"
server_key_iv: "954ad806559d7108e027668a94cf2e92"
server_key_version: "encryptionkey@0"
I20240506 15:40:59.942833 26822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:40:59.943490 26837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:59.943647 26822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:59.943728 26822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "d4517400f4ae4e0faba63e21f081b3c2"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "5d9689b4e61a27568bdd27a54e9078e4"
server_key_iv: "954ad806559d7108e027668a94cf2e92"
server_key_version: "encryptionkey@0"
I20240506 15:40:59.943781 26822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:40:59.962477 26822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:40:59.964686 26822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:40:59.964792 26822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:40:59.964967 26822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:40:59.965266 26822 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:40:59.965301 26822 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:59.965334 26822 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:40:59.965360 26822 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:40:59.970656 26822 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:44201
I20240506 15:40:59.970707 26950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:44201 every 8 connection(s)
I20240506 15:40:59.970969 26822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:40:59.971402 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 26822
I20240506 15:40:59.971514 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:40:59.971740 18371 external_mini_cluster.cc:1428] Setting key 77bca39ecc300d7ca1f70d8f64ba52ce
May 06 15:40:59 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010059, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:40:59.980683 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:59.971808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:59909 (local address 127.17.240.254:43739)
0506 15:40:59.971852 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0506 15:40:59.971855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:40:59.972102 (+ 247us) server_negotiation.cc:407] Connection header received
0506 15:40:59.972762 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:40:59.972764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:40:59.972776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:40:59.972838 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:40:59.973642 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:59.974387 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:40:59.974931 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:40:59.975159 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:59.977647 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:40:59.977651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:40:59.977652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:40:59.977660 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:40:59.979421 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:59.979785 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:59.979786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:59.979787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:59.979812 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:40:59.980032 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:40:59.980033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:40:59.980033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:40:59.980161 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:40:59.980237 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0506 15:40:59.980460 (+ 223us) server_negotiation.cc:299] Negotiation successful
0506 15:40:59.980561 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:40:59.980696 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:40:59.971805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:59909)
0506 15:40:59.971990 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0506 15:40:59.972024 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0506 15:40:59.972632 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:40:59.972897 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:40:59.972900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:40:59.973215 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:40:59.973531 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:59.973536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:59.974486 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:40:59.974487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:40:59.974856 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:40:59.974857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:40:59.975301 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:40:59.975930 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:40:59.975945 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:40:59.977545 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:40:59.979505 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:59.979510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:59.979512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:59.979714 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:59.979901 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:40:59.979902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:40:59.979902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:40:59.979966 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:40:59.980224 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:40:59.980227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:40:59.980335 (+ 108us) client_negotiation.cc:769] Sending connection context
0506 15:40:59.980423 (+ 88us) client_negotiation.cc:240] Negotiation successful
0506 15:40:59.980541 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
I20240506 15:40:59.981166 26951 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43739
I20240506 15:40:59.981279 26951 heartbeater.cc:464] Registering TS with master...
I20240506 15:40:59.981650 26951 heartbeater.cc:511] Master 127.17.240.254:43739 requested a full tablet report, sending...
I20240506 15:40:59.982097 26626 ts_manager.cc:194] Registered new tserver with Master: d4517400f4ae4e0faba63e21f081b3c2 (127.17.240.194:44201)
I20240506 15:40:59.982561 26626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:59909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:41:00.000169 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43739
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:41:00.078233 26956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:00.078431 26956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:00.078467 26956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:00.079829 26956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:00.079879 26956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:00.079916 26956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:41:00.081516 26956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:00.081748 26956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:00.081934 26956 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:00.086784 26961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:00.087122 26964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:00.087631 26962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:00.088171 26956 server_base.cc:1034] running on GCE node
I20240506 15:41:00.088330 26956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:00.088521 26956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:00.089665 26956 hybrid_clock.cc:648] HybridClock initialized: now 1715010060089653 us; error 23 us; skew 500 ppm
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010060, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:41:00.092015 26956 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:41:00.092420 26956 webserver.cc:460] Webserver started at http://127.17.240.195:41069/ using document root <none> and password file <none>
I20240506 15:41:00.092607 26956 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:00.092658 26956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:00.092778 26956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:00.093760 26956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "eb19b34c97974d969372a48f830d4337"
format_stamp: "Formatted at 2024-05-06 15:41:00 on dist-test-slave-v0k1"
server_key: "05c150f00963c1b43010f29b99dfc7bd"
server_key_iv: "f13a27eb5ed9ac4f56d5df2220825076"
server_key_version: "encryptionkey@0"
I20240506 15:41:00.094084 26956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "eb19b34c97974d969372a48f830d4337"
format_stamp: "Formatted at 2024-05-06 15:41:00 on dist-test-slave-v0k1"
server_key: "05c150f00963c1b43010f29b99dfc7bd"
server_key_iv: "f13a27eb5ed9ac4f56d5df2220825076"
server_key_version: "encryptionkey@0"
I20240506 15:41:00.095281 26956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:41:00.095927 26971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.096076 26956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.096153 26956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "eb19b34c97974d969372a48f830d4337"
format_stamp: "Formatted at 2024-05-06 15:41:00 on dist-test-slave-v0k1"
server_key: "05c150f00963c1b43010f29b99dfc7bd"
server_key_iv: "f13a27eb5ed9ac4f56d5df2220825076"
server_key_version: "encryptionkey@0"
I20240506 15:41:00.096212 26956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:00.111953 26956 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:00.113691 26956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:00.113806 26956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:00.113999 26956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:00.114288 26956 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:00.114338 26956 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.114367 26956 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:00.114387 26956 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.120170 26956 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:36071
I20240506 15:41:00.120205 27084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:36071 every 8 connection(s)
I20240506 15:41:00.120527 26956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:41:00.124642 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 26956
I20240506 15:41:00.124727 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:41:00.125003 18371 external_mini_cluster.cc:1428] Setting key 2feb7ada2349eb9e1a3ad8b1b3f5ed97
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010060, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:41:00.128942 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.121251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:35619 (local address 127.17.240.254:43739)
0506 15:41:00.121309 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.121311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.121489 (+ 178us) server_negotiation.cc:407] Connection header received
0506 15:41:00.121995 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.121996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.122005 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.122056 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.122807 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.123390 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.123923 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.124036 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.126409 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:00.126412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:00.126412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:41:00.126419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:00.127867 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:00.128202 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:00.128203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:00.128203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:00.128225 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:00.128391 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:00.128392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:00.128393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:00.128514 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:00.128586 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:00.128801 (+ 215us) server_negotiation.cc:299] Negotiation successful
0506 15:41:00.128872 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240506 15:41:00.129127 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.121242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:35619)
0506 15:41:00.121391 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:00.121420 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:00.121885 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:00.122109 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:00.122112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:00.122411 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:00.122719 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.122723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.123455 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.123455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:00.123864 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.123865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.124165 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.124686 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:00.124710 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:00.126307 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:41:00.127943 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:41:00.127947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:41:00.127949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:41:00.128133 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:41:00.128276 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:41:00.128277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:41:00.128277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:41:00.128319 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:41:00.128582 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:41:00.128586 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:41:00.128697 (+ 111us) client_negotiation.cc:769] Sending connection context
0506 15:41:00.128783 (+ 86us) client_negotiation.cc:240] Negotiation successful
0506 15:41:00.128886 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20240506 15:41:00.129576 27085 heartbeater.cc:347] Connected to a master server at 127.17.240.254:43739
I20240506 15:41:00.129663 27085 heartbeater.cc:464] Registering TS with master...
I20240506 15:41:00.129848 27085 heartbeater.cc:511] Master 127.17.240.254:43739 requested a full tablet report, sending...
I20240506 15:41:00.130268 26626 ts_manager.cc:194] Registered new tserver with Master: eb19b34c97974d969372a48f830d4337 (127.17.240.195:36071)
I20240506 15:41:00.130648 26626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:35619
I20240506 15:41:00.136843 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab.
Entry for principal oryx/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab.
WARNING: no policy specified for oryx/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab.
Entry for principal oryx/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab.
WARNING: no policy specified for oryx/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab.
Entry for principal oryx/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab.
I20240506 15:41:00.210037 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 26688
I20240506 15:41:00.217037 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 26822
I20240506 15:41:00.224090 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 26956
I20240506 15:41:00.229322 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 26594
I20240506 15:41:00.235484 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43739
--webserver_interface=127.17.240.254
--webserver_port=33079
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:43739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:41:00.318639 27094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:00.318819 27094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:00.318854 27094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:00.320729 27094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:41:00.320784 27094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:00.320806 27094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:00.320827 27094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:41:00.320859 27094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:41:00.322856 27094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:43739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=33079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:00.323082 27094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:00.323254 27094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:00.327541 27102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:00.328075 27099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:00.328547 27100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:00.329218 27094 server_base.cc:1034] running on GCE node
I20240506 15:41:00.329409 27094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:00.329629 27094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:00.330750 27094 hybrid_clock.cc:648] HybridClock initialized: now 1715010060330734 us; error 32 us; skew 500 ppm
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010060, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:41:00.333048 27094 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:41:00.333420 27094 webserver.cc:460] Webserver started at http://127.17.240.254:33079/ using document root <none> and password file <none>
I20240506 15:41:00.333595 27094 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:00.333652 27094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:00.334815 27094 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:41:00.335393 27109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.335536 27094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.335621 27094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "936d030ee620b69994f2a41cab02b123"
server_key_iv: "c2a429a4ad0be613ac43cf8f8ff23280"
server_key_version: "encryptionkey@0"
I20240506 15:41:00.335876 27094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:00.342955 27094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:00.344853 27094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:00.344966 27094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:00.348598 27094 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:43739
I20240506 15:41:00.348647 27161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:43739 every 8 connection(s)
I20240506 15:41:00.348955 27094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:41:00.349547 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 27094
I20240506 15:41:00.349864 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:37613
--local_ip_for_outbound_sockets=127.17.240.193
--tserver_master_addrs=127.17.240.254:43739
--webserver_port=37969
--webserver_interface=127.17.240.193
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.240.193
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:41:00.350731 27162 sys_catalog.cc:263] Verifying existing consensus state
I20240506 15:41:00.351647 27162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8: Bootstrap starting.
I20240506 15:41:00.360893 27162 log.cc:824] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8: Log is configured to *not* fsync() on all Append() calls
I20240506 15:41:00.362671 27162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:41:00.362951 27162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8: Bootstrap complete.
I20240506 15:41:00.365116 27162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } }
I20240506 15:41:00.365310 27162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab4c2274c3b4ceeaf6e06122e0818a8, State: Initialized, Role: FOLLOWER
I20240506 15:41:00.365427 27162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } }
I20240506 15:41:00.365489 27162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:41:00.365526 27162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:41:00.365571 27162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:41:00.366454 27162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } }
I20240506 15:41:00.366542 27162 leader_election.cc:302] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ab4c2274c3b4ceeaf6e06122e0818a8; no voters:
I20240506 15:41:00.366712 27162 leader_election.cc:288] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240506 15:41:00.366770 27166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:41:00.366894 27166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [term 2 LEADER]: Becoming Leader. State: Replica: 8ab4c2274c3b4ceeaf6e06122e0818a8, State: Running, Role: LEADER
I20240506 15:41:00.366945 27162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:41:00.366973 27166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } }
I20240506 15:41:00.367271 27166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ab4c2274c3b4ceeaf6e06122e0818a8. Latest consensus state: current_term: 2 leader_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } } }
I20240506 15:41:00.367353 27168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab4c2274c3b4ceeaf6e06122e0818a8" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43739 } } }
I20240506 15:41:00.367354 27166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [sys.catalog]: This master's current role is: LEADER
I20240506 15:41:00.367457 27168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8 [sys.catalog]: This master's current role is: LEADER
I20240506 15:41:00.367728 27172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:41:00.368335 27172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:41:00.368468 27172 catalog_manager.cc:1259] Loaded cluster ID: a8f6ed60097440dea1a5618325ccac95
I20240506 15:41:00.368721 27172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:41:00.369550 27172 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:41:00.369704 27172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ab4c2274c3b4ceeaf6e06122e0818a8: Loaded TSK: 0
I20240506 15:41:00.369881 27172 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240506 15:41:00.444268 27164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:00.444450 27164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:00.444483 27164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:00.445780 27164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:00.445824 27164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:00.445861 27164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:41:00.447460 27164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.193.keytab
--principal=oryx/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:37613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=37969
--tserver_master_addrs=127.17.240.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:00.447731 27164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:00.447914 27164 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:00.452380 27189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:00.452805 27188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:00.453346 27191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:00.453943 27164 server_base.cc:1034] running on GCE node
I20240506 15:41:00.454098 27164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:00.454295 27164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:00.455420 27164 hybrid_clock.cc:648] HybridClock initialized: now 1715010060455407 us; error 29 us; skew 500 ppm
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010060, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:41:00.463280 27164 init.cc:377] Logged in from keytab as oryx/127.17.240.193@KRBTEST.COM (short username oryx)
I20240506 15:41:00.463714 27164 webserver.cc:460] Webserver started at http://127.17.240.193:37969/ using document root <none> and password file <none>
I20240506 15:41:00.463893 27164 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:00.463943 27164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:00.465207 27164 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:41:00.465708 27198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.465885 27164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:41:00.465950 27164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "4f703877c5b841e79c40bf07016b2136"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "51e41c22172f278b5bc808b965778a6f"
server_key_iv: "c78e016081aff9652b29938c65f68a01"
server_key_version: "encryptionkey@0"
I20240506 15:41:00.466248 27164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:00.495888 27164 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:00.497906 27164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:00.498006 27164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:00.498243 27164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:00.498553 27164 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:00.498585 27164 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.498615 27164 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:00.498631 27164 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.505537 27164 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:37613
I20240506 15:41:00.505610 27311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:37613 every 8 connection(s)
I20240506 15:41:00.505905 27164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:00.513137 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.506560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:56833)
0506 15:41:00.506710 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:00.506738 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:00.507177 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:00.507584 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:00.507588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:00.507887 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:00.508165 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.508169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.509624 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.509625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:00.509985 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.509986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.510308 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.510893 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:00.510906 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:00.513014 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
W20240506 15:41:00.513309 27315 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:00.506652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:56833 (local address 127.17.240.254:43739)
0506 15:41:00.506824 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.506835 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.506862 (+ 27us) server_negotiation.cc:407] Connection header received
0506 15:41:00.507358 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.507364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.507410 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.507576 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.508275 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.509522 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.510059 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.510291 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.513164 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:56833: BlockingRecv error: recv got EOF from 127.17.240.193:56833 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
W20240506 15:41:00.513600 27312 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:43739 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.240.254:43739: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
I20240506 15:41:00.515074 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 27164
I20240506 15:41:00.515384 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:44201
--local_ip_for_outbound_sockets=127.17.240.194
--tserver_master_addrs=127.17.240.254:43739
--webserver_port=43439
--webserver_interface=127.17.240.194
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.240.194
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:00.517741 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.513820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:44681)
0506 15:41:00.513869 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:00.513881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:00.513927 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:00.514059 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:00.514060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:00.514206 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:00.514412 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.514414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.515026 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.515027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:00.515436 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.515439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.515513 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.516024 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:00.516028 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:00.517671 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20240506 15:41:00.517820 27315 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:00.513873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:44681 (local address 127.17.240.254:43739)
0506 15:41:00.513917 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.513918 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.513927 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:00.513991 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.513992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.514000 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.514044 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.514481 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.514934 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.515671 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.515793 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.517777 (+ 1984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:44681: BlockingRecv error: recv got EOF from 127.17.240.193:44681 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:00.522805 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.519274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:57639)
0506 15:41:00.519360 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:00.519371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:00.519415 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:00.519540 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:00.519541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:00.519665 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:00.519781 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.519782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.520373 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.520374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:00.520703 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.520705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.520765 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.521255 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:00.521259 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:00.522726 (+ 1467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20240506 15:41:00.522876 27315 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:00.519323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:57639 (local address 127.17.240.254:43739)
0506 15:41:00.519363 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.519365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.519401 (+ 36us) server_negotiation.cc:407] Connection header received
0506 15:41:00.519474 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.519474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.519483 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.519532 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.519847 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.520305 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.520769 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.520938 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.522811 (+ 1873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:57639: BlockingRecv error: recv got EOF from 127.17.240.193:57639 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20240506 15:41:00.523231 27312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240506 15:41:00.604035 27316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:00.604218 27316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:00.604243 27316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:00.605513 27316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:00.605553 27316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:00.605582 27316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:41:00.607169 27316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.194.keytab
--principal=oryx/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:44201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=43439
--tserver_master_addrs=127.17.240.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:00.607343 27316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:00.607511 27316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:00.612417 27321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:00.612829 27322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:00.613389 27324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:00.614023 27316 server_base.cc:1034] running on GCE node
I20240506 15:41:00.614183 27316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:00.614364 27316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:00.615486 27316 hybrid_clock.cc:648] HybridClock initialized: now 1715010060615470 us; error 34 us; skew 500 ppm
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010060, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:41:00.617813 27316 init.cc:377] Logged in from keytab as oryx/127.17.240.194@KRBTEST.COM (short username oryx)
I20240506 15:41:00.618183 27316 webserver.cc:460] Webserver started at http://127.17.240.194:43439/ using document root <none> and password file <none>
I20240506 15:41:00.618346 27316 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:00.618393 27316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:00.619593 27316 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:41:00.620077 27331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.620241 27316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.620307 27316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "d4517400f4ae4e0faba63e21f081b3c2"
format_stamp: "Formatted at 2024-05-06 15:40:59 on dist-test-slave-v0k1"
server_key: "5d9689b4e61a27568bdd27a54e9078e4"
server_key_iv: "954ad806559d7108e027668a94cf2e92"
server_key_version: "encryptionkey@0"
I20240506 15:41:00.620571 27316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:00.635737 27316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:00.637701 27316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:00.637853 27316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:00.638025 27316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:00.638312 27316 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:00.638345 27316 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.638377 27316 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:00.638403 27316 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.643321 27316 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:44201
I20240506 15:41:00.643358 27444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:44201 every 8 connection(s)
I20240506 15:41:00.643641 27316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:00.649397 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.644230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:60605)
0506 15:41:00.644366 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:00.644393 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:00.644815 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:00.645021 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:00.645024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:00.645304 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:00.645575 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.645578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.646184 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.646185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:00.646556 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.646557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.646845 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.647362 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:00.647377 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:00.649277 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
W20240506 15:41:00.649432 27315 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:00.644242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:60605 (local address 127.17.240.254:43739)
0506 15:41:00.644299 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.644301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.644468 (+ 167us) server_negotiation.cc:407] Connection header received
0506 15:41:00.644912 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.644912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.644925 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.644968 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.645651 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.646124 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.646610 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.646712 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.649387 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:60605: BlockingRecv error: recv got EOF from 127.17.240.194:60605 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20240506 15:41:00.649609 27445 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:43739 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.240.254:43739: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
I20240506 15:41:00.651538 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 27316
I20240506 15:41:00.651856 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:36071
--local_ip_for_outbound_sockets=127.17.240.195
--tserver_master_addrs=127.17.240.254:43739
--webserver_port=41069
--webserver_interface=127.17.240.195
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.240.195
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:00.653734 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.649800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:55747)
0506 15:41:00.649849 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:00.649861 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:00.649908 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:00.650038 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:00.650039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:00.650154 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:00.650295 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.650297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.650888 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.650889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:00.651274 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.651276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.651332 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.651862 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:00.651865 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:00.653652 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20240506 15:41:00.653774 27315 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:00.649854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:55747 (local address 127.17.240.254:43739)
0506 15:41:00.649908 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.649910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.649920 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:41:00.649968 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.649968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.649979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.650019 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.650366 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.650814 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.651355 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.651535 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.653730 (+ 2195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:55747: BlockingRecv error: recv got EOF from 127.17.240.194:55747 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:00.657886 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.654088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:36315)
0506 15:41:00.654128 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:00.654139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:00.654183 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:00.654308 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:00.654308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:00.654407 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:00.654514 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.654515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.655420 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.655421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:00.655762 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.655764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.655821 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.656202 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:00.656206 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:00.657843 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20240506 15:41:00.658027 27315 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:00.654146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:36315 (local address 127.17.240.254:43739)
0506 15:41:00.654199 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.654201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.654222 (+ 21us) server_negotiation.cc:407] Connection header received
0506 15:41:00.654242 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.654243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.654254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.654295 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.654571 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.655347 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.655834 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.656031 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.657950 (+ 1919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:36315: BlockingRecv error: recv got EOF from 127.17.240.194:36315 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20240506 15:41:00.658182 27445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240506 15:41:00.731133 27448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:00.731308 27448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:00.731330 27448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:00.732632 27448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:00.732676 27448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:00.732705 27448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:41:00.734297 27448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/krb5kdc/oryx_127.17.240.195.keytab
--principal=oryx/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:36071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=41069
--tserver_master_addrs=127.17.240.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:00.734491 27448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:00.734660 27448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:00.739475 27456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:00.739797 27454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:00.740314 27453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:00.740762 27448 server_base.cc:1034] running on GCE node
I20240506 15:41:00.740918 27448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:00.741129 27448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:00.742269 27448 hybrid_clock.cc:648] HybridClock initialized: now 1715010060742251 us; error 28 us; skew 500 ppm
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010060, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:41:00.744616 27448 init.cc:377] Logged in from keytab as oryx/127.17.240.195@KRBTEST.COM (short username oryx)
I20240506 15:41:00.744988 27448 webserver.cc:460] Webserver started at http://127.17.240.195:41069/ using document root <none> and password file <none>
I20240506 15:41:00.745158 27448 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:00.745218 27448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:00.746394 27448 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:41:00.746862 27463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.747022 27448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.747105 27448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "eb19b34c97974d969372a48f830d4337"
format_stamp: "Formatted at 2024-05-06 15:41:00 on dist-test-slave-v0k1"
server_key: "05c150f00963c1b43010f29b99dfc7bd"
server_key_iv: "f13a27eb5ed9ac4f56d5df2220825076"
server_key_version: "encryptionkey@0"
I20240506 15:41:00.747381 27448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:00.768354 27448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:00.770664 27448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:00.770795 27448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:00.771041 27448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:00.771342 27448 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:00.771375 27448 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.771409 27448 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:00.771446 27448 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:00.776510 27448 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:36071
I20240506 15:41:00.776566 27576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:36071 every 8 connection(s)
I20240506 15:41:00.776859 27448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:00.782727 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.777478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:33527)
0506 15:41:00.777620 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:00.777644 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:00.778088 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:00.778276 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:00.778279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:00.778567 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:00.778845 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.778849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.779506 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.779507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:00.779892 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.779894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.780218 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.780700 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:00.780712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:00.782609 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
W20240506 15:41:00.782748 27315 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:00.777506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:33527 (local address 127.17.240.254:43739)
0506 15:41:00.777593 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.777595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.777720 (+ 125us) server_negotiation.cc:407] Connection header received
0506 15:41:00.778170 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.778171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.778181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.778232 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.778913 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.779395 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.779964 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.780065 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.782702 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:33527: BlockingRecv error: recv got EOF from 127.17.240.195:33527 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20240506 15:41:00.782979 27577 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:43739 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.240.254:43739: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:00.786659 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.783232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:35511)
0506 15:41:00.783275 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:00.783284 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:00.783322 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:00.783516 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:00.783517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:00.783640 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:00.783777 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.783778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.784365 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.784365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:00.784736 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.784738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.784799 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.785189 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:00.785193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:00.786614 (+ 1421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20240506 15:41:00.786759 27315 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:00.783375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:35511 (local address 127.17.240.254:43739)
0506 15:41:00.783426 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.783427 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.783434 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:41:00.783452 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.783453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.783460 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.783505 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.783844 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.784300 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.784803 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.784975 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.786698 (+ 1723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:35511: BlockingRecv error: recv got EOF from 127.17.240.195:35511 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20240506 15:41:00.786901 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 27448
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:00 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:00.790609 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.787090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:48485)
0506 15:41:00.787264 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:00.787272 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:00.787315 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:00.787483 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:00.787484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:00.787626 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:00.787792 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.787794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.788347 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.788348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:00.788691 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:00.788692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.788744 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.789124 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:00.789127 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:00.790568 (+ 1441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162}
W20240506 15:41:00.790709 27315 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:00.787161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:48485 (local address 127.17.240.254:43739)
0506 15:41:00.787207 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.787209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.787318 (+ 109us) server_negotiation.cc:407] Connection header received
0506 15:41:00.787374 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.787375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.787383 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.787438 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.787864 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.788283 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.788749 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.788911 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.790647 (+ 1736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:48485: BlockingRecv error: recv got EOF from 127.17.240.195:48485 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20240506 15:41:00.790714 27577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240506 15:41:00.793288 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:00.787260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.17.240.254:43739)
0506 15:41:00.787424 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:00.787425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:00.787496 (+ 71us) server_negotiation.cc:407] Connection header received
0506 15:41:00.787566 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:00.787566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:00.787573 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:00.787612 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:00.788341 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.788795 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:00.789395 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:00.789576 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:00.790545 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:00.790551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:00.790552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:41:00.790583 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:00.792176 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:00.792469 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:00.792472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:00.792472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:00.792520 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:00.792660 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:00.792660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:00.792661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:00.792827 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:00.792951 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:00.793073 (+ 122us) server_negotiation.cc:299] Negotiation successful
0506 15:41:00.793200 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
May 06 15:41:01 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:01 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:01.527734 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:01.523588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:53385)
0506 15:41:01.523730 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:01.523742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:01.523807 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:01.523964 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:01.523965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:01.524166 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:01.524348 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:01.524350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.525021 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:01.525021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:01.525456 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:01.525459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.525517 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:01.525995 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:01.525999 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:01.527648 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
W20240506 15:41:01.527820 27583 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:01.523641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:53385 (local address 127.17.240.254:43739)
0506 15:41:01.523765 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:01.523767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:01.523790 (+ 23us) server_negotiation.cc:407] Connection header received
0506 15:41:01.523856 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:01.523857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:01.523867 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:01.523957 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:01.524431 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.524958 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:01.525516 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.525631 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:01.527754 (+ 2123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:53385: BlockingRecv error: recv got EOF from 127.17.240.193:53385 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
May 06 15:41:01 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:01 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:01.662485 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:01.658548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:55033)
0506 15:41:01.658681 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:01.658691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:01.658749 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:01.658855 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:01.658856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:01.659035 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:01.659219 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:01.659221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.659841 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:01.659842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:01.660225 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:01.660226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.660283 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:01.660802 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:01.660806 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:01.662421 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
W20240506 15:41:01.662564 27583 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:01.658571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:55033 (local address 127.17.240.254:43739)
0506 15:41:01.658635 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:01.658637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:01.658745 (+ 108us) server_negotiation.cc:407] Connection header received
0506 15:41:01.658792 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:01.658793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:01.658803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:01.658850 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:01.659279 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.659779 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:01.660317 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.660487 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:01.662516 (+ 2029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:55033: BlockingRecv error: recv got EOF from 127.17.240.194:55033 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
May 06 15:41:01 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:01 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:01.795259 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:01.791121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:60041)
0506 15:41:01.791255 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:01.791268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:01.791338 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:01.791512 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:01.791513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:01.791680 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:01.791859 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:01.791860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.792500 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:01.792501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:01.792881 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:01.792883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.792946 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:01.793454 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:01.793458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:01.795193 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
W20240506 15:41:01.795351 27583 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:01.791215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:60041 (local address 127.17.240.254:43739)
0506 15:41:01.791279 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:01.791281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:01.791321 (+ 40us) server_negotiation.cc:407] Connection header received
0506 15:41:01.791397 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:01.791398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:01.791412 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:01.791467 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:01.791940 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.792424 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:01.792956 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:01.793160 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:01.795282 (+ 2122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:60041: BlockingRecv error: recv got EOF from 127.17.240.195:60041 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
May 06 15:41:02 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:02 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:02.532336 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:02.528275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:35003)
0506 15:41:02.528399 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:02.528409 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:02.528471 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:02.528603 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:02.528604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:02.528773 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:02.528931 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:02.528932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.529681 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:02.529683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:02.530137 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:02.530139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.530194 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:02.530682 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:02.530685 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:02.532275 (+ 1590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
W20240506 15:41:02.532450 27587 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:02.528343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:35003 (local address 127.17.240.254:43739)
0506 15:41:02.528446 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:02.528448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:02.528458 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:41:02.528526 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:02.528527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:02.528537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:02.528594 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:02.528999 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.529593 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:02.530198 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.530329 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:02.532379 (+ 2050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:35003: BlockingRecv error: recv got EOF from 127.17.240.193:35003 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":48,"threads_started":1}
May 06 15:41:02 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:02 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:02.667085 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:02.662972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:38653)
0506 15:41:02.663122 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:02.663132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:02.663194 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:02.663331 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:02.663332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:02.663514 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:02.663680 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:02.663681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.664320 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:02.664321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:02.664691 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:02.664693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.664754 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:02.665295 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:02.665299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:02.666978 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
W20240506 15:41:02.667174 27587 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:02.663057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:38653 (local address 127.17.240.254:43739)
0506 15:41:02.663105 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:02.663107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:02.663194 (+ 87us) server_negotiation.cc:407] Connection header received
0506 15:41:02.663261 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:02.663262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:02.663273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:02.663323 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:02.663760 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.664249 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:02.664761 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.664957 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:02.667114 (+ 2157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:38653: BlockingRecv error: recv got EOF from 127.17.240.194:38653 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
May 06 15:41:02 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:02 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:02.799759 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:02.795721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:37757)
0506 15:41:02.795877 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:02.795891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:02.795959 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:02.796088 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:02.796089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:02.796263 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:02.796426 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:02.796428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.797029 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:02.797030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:02.797396 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:02.797398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.797451 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:02.797961 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:02.797965 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:02.799699 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
W20240506 15:41:02.799902 27587 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:02.795890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:37757 (local address 127.17.240.254:43739)
0506 15:41:02.795955 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:02.795957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:02.795967 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:41:02.796007 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:02.796008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:02.796020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:02.796077 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:02.796484 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.796969 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:02.797455 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:02.797563 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:02.799834 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:37757: BlockingRecv error: recv got EOF from 127.17.240.195:37757 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
May 06 15:41:03 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:03 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:03.536954 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:03.532818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:50869)
0506 15:41:03.532974 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:03.532984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:03.533042 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:03.533185 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:03.533186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:03.533351 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:03.533521 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:03.533522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.534161 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:03.534162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:03.534518 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:03.534519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.534570 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:03.535101 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:03.535105 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:03.536876 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
W20240506 15:41:03.537081 27591 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:03.532897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:50869 (local address 127.17.240.254:43739)
0506 15:41:03.533012 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:03.533013 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:03.533028 (+ 15us) server_negotiation.cc:407] Connection header received
0506 15:41:03.533098 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:03.533098 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:03.533109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:03.533169 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:03.533585 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.534101 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:03.534581 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.534705 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:03.537007 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:50869: BlockingRecv error: recv got EOF from 127.17.240.193:50869 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
May 06 15:41:03 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:03 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:03.671931 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:03.667589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:44999)
0506 15:41:03.667800 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:03.667811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:03.667881 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:03.668005 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:03.668006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:03.668226 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:03.668388 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:03.668390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.669069 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:03.669071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:03.669450 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:03.669452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.669522 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:03.670132 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:03.670137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:03.671850 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":144,"threads_started":1}
W20240506 15:41:03.672019 27591 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:03.667708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:44999 (local address 127.17.240.254:43739)
0506 15:41:03.667765 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:03.667767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:03.667865 (+ 98us) server_negotiation.cc:407] Connection header received
0506 15:41:03.667931 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:03.667932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:03.667944 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:03.667996 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:03.668467 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.668985 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:03.669524 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.669712 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:03.671959 (+ 2247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:44999: BlockingRecv error: recv got EOF from 127.17.240.194:44999 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
May 06 15:41:03 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:03 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:03.804378 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:03.800267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:52803)
0506 15:41:03.800426 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:03.800438 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:03.800509 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:03.800639 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:03.800639 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:03.800822 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:03.800977 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:03.800978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.801638 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:03.801639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:03.802001 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:03.802003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.802066 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:03.802647 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:03.802651 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:03.804298 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
W20240506 15:41:03.804471 27591 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:03.800357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:52803 (local address 127.17.240.254:43739)
0506 15:41:03.800426 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:03.800428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:03.800498 (+ 70us) server_negotiation.cc:407] Connection header received
0506 15:41:03.800565 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:03.800566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:03.800581 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:03.800629 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:03.801055 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.801548 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:03.802078 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:03.802281 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:03.804410 (+ 2129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:52803: BlockingRecv error: recv got EOF from 127.17.240.195:52803 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
May 06 15:41:04 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:04 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:04.541524 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:04.537436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:53019)
0506 15:41:04.537585 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:04.537597 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:04.537663 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:04.537781 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:04.537781 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:04.537951 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:04.538117 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:04.538119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.538775 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:04.538776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:04.539201 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:04.539203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.539272 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:04.539797 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:04.539801 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:04.541462 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
W20240506 15:41:04.541648 27595 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:04.537524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:53019 (local address 127.17.240.254:43739)
0506 15:41:04.537659 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:04.537661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:04.537669 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:41:04.537706 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:04.537707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:04.537716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:04.537772 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:04.538182 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.538692 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:04.539287 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.539415 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:04.541563 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:53019: BlockingRecv error: recv got EOF from 127.17.240.193:53019 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
May 06 15:41:04 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:04 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:04.676600 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:04.672457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:36605)
0506 15:41:04.672603 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:04.672617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:04.672697 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:04.672857 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:04.672858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:04.673040 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:04.673207 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:04.673209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.673870 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:04.673872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:04.674248 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:04.674249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.674313 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:04.674870 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:04.674874 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:04.676514 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
W20240506 15:41:04.676700 27595 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:04.672573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:36605 (local address 127.17.240.254:43739)
0506 15:41:04.672653 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:04.672667 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:04.672678 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:41:04.672766 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:04.672767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:04.672779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:04.672845 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:04.673294 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.673794 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:04.674324 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.674521 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:04.676636 (+ 2115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:36605: BlockingRecv error: recv got EOF from 127.17.240.194:36605 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
May 06 15:41:04 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:04 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:04.809239 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:04.804979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:58955)
0506 15:41:04.805148 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:04.805163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:04.805236 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:04.805377 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:04.805378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:04.805555 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:04.805713 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:04.805715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.806387 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:04.806388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:04.806819 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:04.806822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.806890 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:04.807462 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:04.807466 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:04.809180 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
W20240506 15:41:04.809362 27595 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:04.805114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:58955 (local address 127.17.240.254:43739)
0506 15:41:04.805206 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:04.805208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:04.805220 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:41:04.805294 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:04.805295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:04.805308 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:04.805364 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:04.805778 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.806266 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:04.806928 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:04.807109 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:04.809285 (+ 2176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:58955: BlockingRecv error: recv got EOF from 127.17.240.195:58955 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
May 06 15:41:05 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:05 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:05.551801 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:05.542015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:36539)
0506 15:41:05.542152 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:05.542165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:05.542234 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:05.542385 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:05.542386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:05.542548 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:05.542715 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:05.542717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.543375 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:05.543376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:05.543747 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:05.543748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.543803 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:05.549891 (+ 6088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:05.549896 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:05.551739 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
W20240506 15:41:05.551920 27599 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:05.542094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:36539 (local address 127.17.240.254:43739)
0506 15:41:05.542263 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:05.542265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:05.542274 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:05.542299 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:05.542300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:05.542311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:05.542376 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:05.542782 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.543319 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:05.543851 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.543996 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:05.551849 (+ 7853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:36539: BlockingRecv error: recv got EOF from 127.17.240.193:36539 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
May 06 15:41:05 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:05 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:05.681512 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:05.677140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:55247)
0506 15:41:05.677288 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:05.677298 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:05.677349 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:05.677563 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:05.677565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:05.677740 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:05.677915 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:05.677917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.678700 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:05.678702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:05.679194 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:05.679196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.679257 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:05.679787 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:05.679791 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:05.681453 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
W20240506 15:41:05.681641 27599 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:05.677305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:55247 (local address 127.17.240.254:43739)
0506 15:41:05.677357 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:05.677359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:05.677368 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:05.677398 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:05.677398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:05.677408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:05.677471 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:05.677993 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.678618 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:05.679258 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.679370 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:05.681570 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:55247: BlockingRecv error: recv got EOF from 127.17.240.194:55247 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
May 06 15:41:05 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:05 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:05.813822 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:05.809699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:52931)
0506 15:41:05.809837 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:05.809849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:05.809916 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:05.810046 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:05.810047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:05.810230 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:05.810384 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:05.810386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.811075 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:05.811077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:05.811461 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:05.811462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.811526 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:05.812076 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:05.812081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:05.813758 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
W20240506 15:41:05.813932 27599 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:05.809809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:52931 (local address 127.17.240.254:43739)
0506 15:41:05.809872 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:05.809874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:05.809898 (+ 24us) server_negotiation.cc:407] Connection header received
0506 15:41:05.809973 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:05.809974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:05.809988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:05.810035 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:05.810470 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.810966 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:05.811564 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:05.811759 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:05.813855 (+ 2096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:52931: BlockingRecv error: recv got EOF from 127.17.240.195:52931 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
May 06 15:41:06 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:06 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:06.556233 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:06.552247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:54527)
0506 15:41:06.552392 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:06.552401 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:06.552449 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:06.552581 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:06.552582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:06.552746 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:06.552902 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:06.552903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.553569 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:06.553570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:06.553941 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:06.553943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.553995 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:06.554517 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:06.554521 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:06.556156 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
W20240506 15:41:06.556336 27603 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:06.552313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:54527 (local address 127.17.240.254:43739)
0506 15:41:06.552447 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:06.552449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:06.552458 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:06.552497 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:06.552498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:06.552516 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:06.552566 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:06.552961 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.553468 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:06.554000 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.554113 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:06.556256 (+ 2143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:54527: BlockingRecv error: recv got EOF from 127.17.240.193:54527 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
May 06 15:41:06 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:06 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:06.686137 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:06.682019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:32933)
0506 15:41:06.682184 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:06.682195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:06.682273 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:06.682407 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:06.682408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:06.682596 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:06.682751 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:06.682752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.683453 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:06.683455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:06.683841 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:06.683843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.683907 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:06.684414 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:06.684418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:06.686015 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
W20240506 15:41:06.686188 27603 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:06.682099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:32933 (local address 127.17.240.254:43739)
0506 15:41:06.682184 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:06.682186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:06.682265 (+ 79us) server_negotiation.cc:407] Connection header received
0506 15:41:06.682328 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:06.682329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:06.682342 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:06.682395 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:06.682833 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.683377 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:06.683917 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.684116 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:06.686127 (+ 2011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:32933: BlockingRecv error: recv got EOF from 127.17.240.194:32933 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
May 06 15:41:06 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:06 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:06.818326 27605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:06.814326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:56181)
0506 15:41:06.814501 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:06.814511 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:06.814585 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:06.814704 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:06.814704 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:06.814873 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:06.815080 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:06.815082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.815704 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:06.815706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:06.816091 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:06.816093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.816149 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:06.816649 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:06.816653 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:06.818212 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
W20240506 15:41:06.818387 27603 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:06.814405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:56181 (local address 127.17.240.254:43739)
0506 15:41:06.814470 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:06.814472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:06.814577 (+ 105us) server_negotiation.cc:407] Connection header received
0506 15:41:06.814630 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:06.814631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:06.814643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:06.814689 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:06.815139 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.815643 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:06.816152 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:06.816253 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:06.818323 (+ 2070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:56181: BlockingRecv error: recv got EOF from 127.17.240.195:56181 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
May 06 15:41:07 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:07 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:07.561272 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:07.556758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:42231)
0506 15:41:07.556916 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:07.556926 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:07.556985 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:07.557181 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:07.557182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:07.557388 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:07.557593 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:07.557596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.558356 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:07.558357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:07.558735 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:07.558737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.558805 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:07.559412 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:07.559416 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:07.561211 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
W20240506 15:41:07.561388 27607 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:07.556873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:42231 (local address 127.17.240.254:43739)
0506 15:41:07.557045 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:07.557047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:07.557057 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:41:07.557085 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:07.557085 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:07.557096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:07.557163 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:07.557681 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.558254 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:07.558818 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.559042 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:07.561318 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:42231: BlockingRecv error: recv got EOF from 127.17.240.193:42231 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
May 06 15:41:07 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:07 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:07.690977 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:07.686675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:55767)
0506 15:41:07.686831 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:07.686843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:07.686923 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:07.687097 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:07.687098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:07.687313 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:07.687506 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:07.687509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.688234 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:07.688236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:07.688617 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:07.688619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.688682 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:07.689280 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:07.689284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:07.690917 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
W20240506 15:41:07.691112 27607 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:07.686821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:55767 (local address 127.17.240.254:43739)
0506 15:41:07.686911 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:07.686913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:07.686922 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:07.687000 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:07.687001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:07.687034 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:07.687087 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:07.687620 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.688162 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:07.688691 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.688892 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:07.691042 (+ 2150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:55767: BlockingRecv error: recv got EOF from 127.17.240.194:55767 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
May 06 15:41:07 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:07 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:07.823035 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:07.818881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:45173)
0506 15:41:07.819058 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:07.819071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:07.819142 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:07.819299 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:07.819300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:07.819472 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:07.819640 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:07.819642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.820266 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:07.820267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:07.820636 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:07.820637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.820698 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:07.821215 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:07.821220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:07.822929 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
W20240506 15:41:07.823132 27607 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:07.818959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:45173 (local address 127.17.240.254:43739)
0506 15:41:07.819067 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:07.819070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:07.819135 (+ 65us) server_negotiation.cc:407] Connection header received
0506 15:41:07.819206 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:07.819207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:07.819220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:07.819292 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:07.819719 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.820197 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:07.820711 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:07.820894 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:07.823075 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:45173: BlockingRecv error: recv got EOF from 127.17.240.195:45173 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
May 06 15:41:08 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:08 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:08.566015 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:08.561796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:54193)
0506 15:41:08.561967 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:08.561978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:08.562038 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:08.562214 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:08.562215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:08.562374 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:08.562543 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:08.562544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.563244 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:08.563245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:08.563653 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:08.563654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.563717 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:08.564266 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:08.564270 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:08.565947 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
W20240506 15:41:08.566131 27611 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:08.561949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:54193 (local address 127.17.240.254:43739)
0506 15:41:08.562094 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:08.562096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:08.562103 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:41:08.562125 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:08.562126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:08.562135 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:08.562197 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:08.562607 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.563168 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:08.563722 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.563829 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:08.566059 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:54193: BlockingRecv error: recv got EOF from 127.17.240.193:54193 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
May 06 15:41:08 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:08 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:08.695895 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:08.691538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:49185)
0506 15:41:08.691671 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:08.691685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:08.691758 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:08.691905 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:08.691906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:08.692081 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:08.692234 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:08.692236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.693062 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:08.693064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:08.693430 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:08.693432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.693497 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:08.694047 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:08.694051 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:08.695805 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
W20240506 15:41:08.695991 27611 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:08.691631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:49185 (local address 127.17.240.254:43739)
0506 15:41:08.691705 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:08.691707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:08.691739 (+ 32us) server_negotiation.cc:407] Connection header received
0506 15:41:08.691826 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:08.691827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:08.691842 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:08.691892 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:08.692322 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.692975 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:08.693509 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.693696 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:08.695931 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:49185: BlockingRecv error: recv got EOF from 127.17.240.194:49185 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
May 06 15:41:08 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:08 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:08.827626 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:08.823590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:35093)
0506 15:41:08.823726 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:08.823736 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:08.823795 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:08.823930 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:08.823930 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:08.824095 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:08.824253 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:08.824254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.824889 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:08.824891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:08.825297 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:08.825298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.825364 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:08.825902 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:08.825907 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:08.827570 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
W20240506 15:41:08.827786 27611 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:08.823657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:35093 (local address 127.17.240.254:43739)
0506 15:41:08.823722 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:08.823724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:08.823791 (+ 67us) server_negotiation.cc:407] Connection header received
0506 15:41:08.823838 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:08.823839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:08.823852 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:08.823905 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:08.824315 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.824796 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:08.825372 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:08.825487 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:08.827668 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:35093: BlockingRecv error: recv got EOF from 127.17.240.195:35093 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
May 06 15:41:09 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:09 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:09.571220 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:09.566490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:39159)
0506 15:41:09.566626 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:09.566637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:09.566683 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:09.566878 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:09.566879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:09.567112 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:09.567281 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:09.567283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.568325 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:09.568326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:09.568796 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:09.568798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.568859 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:09.569396 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:09.569400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:09.571158 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
W20240506 15:41:09.571337 27615 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:09.566525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:39159 (local address 127.17.240.254:43739)
0506 15:41:09.566666 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:09.566668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:09.566678 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:41:09.566747 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:09.566747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:09.566758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:09.566816 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:09.567367 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.568257 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:09.568864 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.568987 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:09.571264 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:39159: BlockingRecv error: recv got EOF from 127.17.240.193:39159 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
May 06 15:41:09 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:09 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:09.700495 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:09.696360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:48033)
0506 15:41:09.696524 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:09.696537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:09.696610 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:09.696748 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:09.696749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:09.696921 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:09.697073 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:09.697075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.697717 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:09.697719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:09.698098 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:09.698099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.698166 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:09.698692 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:09.698696 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:09.700374 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
W20240506 15:41:09.700546 27615 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:09.696443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:48033 (local address 127.17.240.254:43739)
0506 15:41:09.696527 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:09.696530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:09.696597 (+ 67us) server_negotiation.cc:407] Connection header received
0506 15:41:09.696675 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:09.696675 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:09.696690 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:09.696741 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:09.697151 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.697644 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:09.698174 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.698373 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:09.700481 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:48033: BlockingRecv error: recv got EOF from 127.17.240.194:48033 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
May 06 15:41:09 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:09 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:09.832621 27617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:09.828124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:44883)
0506 15:41:09.828283 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:09.828293 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:09.828360 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:09.828546 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:09.828547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:09.828735 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:09.828891 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:09.828893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.829580 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:09.829581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:09.829965 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:09.829967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.830026 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:09.830600 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:09.830604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:09.832540 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":138,"threads_started":1}
W20240506 15:41:09.832717 27615 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:09.828200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:44883 (local address 127.17.240.254:43739)
0506 15:41:09.828296 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:09.828298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:09.828359 (+ 61us) server_negotiation.cc:407] Connection header received
0506 15:41:09.828402 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:09.828403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:09.828416 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:09.828504 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:09.828966 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.829501 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:09.830042 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:09.830164 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:09.832653 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:44883: BlockingRecv error: recv got EOF from 127.17.240.195:44883 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
May 06 15:41:10 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:10 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:10.582551 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:10.571744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:46845)
0506 15:41:10.571920 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:10.571933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:10.572011 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:10.572149 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:10.572150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:10.572314 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:10.572484 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:10.572486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.573150 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:10.573151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:10.573537 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:10.573539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.573611 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:10.580712 (+ 7101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:10.580717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:10.582489 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
W20240506 15:41:10.582723 27619 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:10.571865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:46845 (local address 127.17.240.254:43739)
0506 15:41:10.572003 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:10.572004 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:10.572013 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:10.572059 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:10.572060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:10.572071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:10.572135 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:10.572553 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.573053 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:10.573609 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.573715 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:10.582631 (+ 8916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:46845: BlockingRecv error: recv got EOF from 127.17.240.193:46845 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
May 06 15:41:10 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:10 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:10.705030 27620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:10.700989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:59745)
0506 15:41:10.701128 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:10.701139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:10.701215 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:10.701368 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:10.701369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:10.701549 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:10.701700 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:10.701702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.702354 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:10.702356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:10.702743 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:10.702745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.702809 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:10.703355 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:10.703359 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:10.704950 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
W20240506 15:41:10.705118 27619 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:10.701052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:59745 (local address 127.17.240.254:43739)
0506 15:41:10.701137 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:10.701140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:10.701207 (+ 67us) server_negotiation.cc:407] Connection header received
0506 15:41:10.701285 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:10.701286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:10.701299 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:10.701356 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:10.701780 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.702281 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:10.702819 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.703057 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:10.705057 (+ 2000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:59745: BlockingRecv error: recv got EOF from 127.17.240.194:59745 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
May 06 15:41:10 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:10 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:10.837625 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:10.833110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:52281)
0506 15:41:10.833242 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:10.833252 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:10.833300 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:10.833514 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:10.833515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:10.833714 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:10.833870 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:10.833872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.834556 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:10.834557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:10.834942 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:10.834943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.835039 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:10.835682 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:10.835687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:10.837538 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
W20240506 15:41:10.837723 27619 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:10.833309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:52281 (local address 127.17.240.254:43739)
0506 15:41:10.833377 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:10.833379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:10.833385 (+ 6us) server_negotiation.cc:407] Connection header received
0506 15:41:10.833406 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:10.833406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:10.833443 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:10.833494 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:10.833981 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.834472 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:10.835044 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:10.835236 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:10.837658 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:52281: BlockingRecv error: recv got EOF from 127.17.240.195:52281 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
May 06 15:41:11 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:11 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:11.587317 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:11.583126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:41351)
0506 15:41:11.583247 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:11.583261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:11.583334 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:11.583481 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:11.583481 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:11.583659 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:11.583817 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:11.583818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.584459 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:11.584460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:11.584820 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:11.584821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.584873 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:11.585367 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:11.585371 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:11.587257 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
W20240506 15:41:11.587450 27623 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:11.583233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:41351 (local address 127.17.240.254:43739)
0506 15:41:11.583365 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:11.583367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:11.583376 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:11.583398 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:11.583399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:11.583409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:11.583468 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:11.583878 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.584401 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:11.584885 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.585015 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:11.587358 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:41351: BlockingRecv error: recv got EOF from 127.17.240.193:41351 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
May 06 15:41:11 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:11 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:11.709878 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:11.705581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:58725)
0506 15:41:11.705747 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:11.705758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:11.705827 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:11.705990 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:11.705991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:11.706178 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:11.706343 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:11.706345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.707028 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:11.707029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:11.707398 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:11.707400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.707475 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:11.708004 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:11.708008 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:11.709819 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
W20240506 15:41:11.709992 27623 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:11.705606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:58725 (local address 127.17.240.254:43739)
0506 15:41:11.705668 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:11.705670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:11.705822 (+ 152us) server_negotiation.cc:407] Connection header received
0506 15:41:11.705887 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:11.705888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:11.705899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:11.705965 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:11.706425 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.706919 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:11.707475 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.707676 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:11.709917 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:58725: BlockingRecv error: recv got EOF from 127.17.240.194:58725 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
May 06 15:41:11 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:11 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:11.842190 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:11.838137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:58571)
0506 15:41:11.838291 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:11.838303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:11.838388 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:11.838526 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:11.838527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:11.838712 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:11.838867 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:11.838869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.839566 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:11.839567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:11.839942 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:11.839944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.840011 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:11.840529 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:11.840533 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:11.842110 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
W20240506 15:41:11.842291 27623 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:11.838216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:58571 (local address 127.17.240.254:43739)
0506 15:41:11.838295 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:11.838298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:11.838373 (+ 75us) server_negotiation.cc:407] Connection header received
0506 15:41:11.838442 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:11.838442 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:11.838460 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:11.838510 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:11.838949 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.839491 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:11.840022 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:11.840222 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:11.842228 (+ 2006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:58571: BlockingRecv error: recv got EOF from 127.17.240.195:58571 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
May 06 15:41:12 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:12 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:12.592191 27626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:12.587826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:39219)
0506 15:41:12.587964 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:12.587974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:12.588023 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:12.588170 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:12.588170 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:12.588338 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:12.588502 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:12.588504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.589208 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:12.589209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:12.589562 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:12.589563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.589623 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:12.590147 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:12.590151 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:12.592128 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
W20240506 15:41:12.592304 27627 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:12.587907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:39219 (local address 127.17.240.254:43739)
0506 15:41:12.588043 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:12.588044 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:12.588054 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:41:12.588076 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:12.588077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:12.588087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:12.588157 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:12.588602 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.589137 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:12.589640 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.589767 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:12.592220 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:39219: BlockingRecv error: recv got EOF from 127.17.240.193:39219 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
May 06 15:41:12 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:12 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:12.714651 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:12.710438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:43797)
0506 15:41:12.710618 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:12.710632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:12.710703 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:12.710878 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:12.710878 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:12.711112 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:12.711343 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:12.711345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.712002 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:12.712003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:12.712372 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:12.712373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.712429 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:12.712958 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:12.712962 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:12.714593 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
W20240506 15:41:12.714766 27627 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:12.710549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:43797 (local address 127.17.240.254:43739)
0506 15:41:12.710621 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:12.710623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:12.710697 (+ 74us) server_negotiation.cc:407] Connection header received
0506 15:41:12.710783 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:12.710784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:12.710797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:12.710852 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:12.711435 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.711936 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:12.712486 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.712620 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:12.714699 (+ 2079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:43797: BlockingRecv error: recv got EOF from 127.17.240.194:43797 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
May 06 15:41:12 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:12 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:12.846807 27629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:12.842718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:38129)
0506 15:41:12.842879 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:12.842889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:12.842964 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:12.843129 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:12.843130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:12.843315 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:12.843470 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:12.843472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.844105 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:12.844107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:12.844480 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:12.844482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.844549 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:12.845093 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:12.845098 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:12.846723 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
W20240506 15:41:12.846884 27627 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:12.842789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:38129 (local address 127.17.240.254:43739)
0506 15:41:12.842858 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:12.842860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:12.842959 (+ 99us) server_negotiation.cc:407] Connection header received
0506 15:41:12.843049 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:12.843050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:12.843063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:12.843115 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:12.843548 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.844029 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:12.844554 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:12.844740 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:12.846826 (+ 2086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:38129: BlockingRecv error: recv got EOF from 127.17.240.195:38129 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
May 06 15:41:13 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:13 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:13.597138 27630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:13.592752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:56835)
0506 15:41:13.592938 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:13.592949 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:13.593012 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:13.593169 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:13.593169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:13.593337 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:13.593500 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:13.593502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.594143 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:13.594144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:13.594576 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:13.594578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.594664 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:13.595234 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:13.595241 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:13.597058 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":127,"threads_started":1}
W20240506 15:41:13.597255 27631 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:13.592911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:56835 (local address 127.17.240.254:43739)
0506 15:41:13.593046 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:13.593048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:13.593057 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:13.593083 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:13.593084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:13.593094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:13.593160 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:13.593567 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.594073 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:13.594681 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.594905 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:13.597188 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:56835: BlockingRecv error: recv got EOF from 127.17.240.193:56835 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
May 06 15:41:13 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:13 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:13.719767 27632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:13.715175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:39753)
0506 15:41:13.715297 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:13.715307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:13.715357 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:13.715530 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:13.715532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:13.715705 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:13.715868 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:13.715870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.716783 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:13.716784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:13.717173 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:13.717174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.717247 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:13.717885 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:13.717890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:13.719706 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
W20240506 15:41:13.719949 27631 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:13.715315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:39753 (local address 127.17.240.254:43739)
0506 15:41:13.715368 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:13.715369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:13.715378 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:13.715408 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:13.715408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:13.715418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:13.715485 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:13.715994 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.716695 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:13.717262 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.717450 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:13.719869 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:39753: BlockingRecv error: recv got EOF from 127.17.240.194:39753 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
May 06 15:41:13 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:13 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:13.851833 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:13.847398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:60285)
0506 15:41:13.847592 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:13.847603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:13.847679 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:13.847838 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:13.847839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:13.848010 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:13.848171 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:13.848173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.848853 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:13.848853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:13.849212 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:13.849213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.849265 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:13.849911 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:13.849916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:13.851775 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":121,"threads_started":1}
W20240506 15:41:13.851953 27631 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:13.847506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:60285 (local address 127.17.240.254:43739)
0506 15:41:13.847615 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:13.847617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:13.847679 (+ 62us) server_negotiation.cc:407] Connection header received
0506 15:41:13.847729 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:13.847730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:13.847742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:13.847798 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:13.848248 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.848788 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:13.849266 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:13.849368 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:13.851884 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:60285: BlockingRecv error: recv got EOF from 127.17.240.195:60285 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
May 06 15:41:14 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:14 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:14.602134 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:14.597719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:53523)
0506 15:41:14.597894 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:14.597907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:14.597975 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:14.598105 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:14.598106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:14.598277 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:14.598436 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:14.598438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.599142 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:14.599143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:14.599539 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:14.599541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.599602 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:14.600125 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:14.600128 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:14.602075 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
W20240506 15:41:14.602272 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:14.597826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:53523 (local address 127.17.240.254:43739)
0506 15:41:14.597979 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:14.597980 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:14.597991 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:41:14.598016 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:14.598017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:14.598029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:14.598090 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:14.598504 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.599072 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:14.599614 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.599739 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:14.602189 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:53523: BlockingRecv error: recv got EOF from 127.17.240.193:53523 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
May 06 15:41:14 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:14 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:14.724282 27636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:14.720290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:56601)
0506 15:41:14.720415 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:14.720425 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:14.720486 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:14.720615 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:14.720615 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:14.720779 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:14.720937 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:14.720938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.721563 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:14.721564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:14.721921 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:14.721922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.721993 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:14.722597 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:14.722601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:14.724200 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
W20240506 15:41:14.724361 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:14.720349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:56601 (local address 127.17.240.254:43739)
0506 15:41:14.720419 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:14.720421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:14.720487 (+ 66us) server_negotiation.cc:407] Connection header received
0506 15:41:14.720535 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:14.720536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:14.720547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:14.720607 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:14.721011 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.721499 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:14.722009 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.722203 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:14.724303 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:56601: BlockingRecv error: recv got EOF from 127.17.240.194:56601 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
May 06 15:41:14 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:14 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.195@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:14.856654 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:14.852353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.195:38875)
0506 15:41:14.852524 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:14.852536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:14.852629 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:14.852850 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:14.852851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:14.853032 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:14.853200 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:14.853202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.853846 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:14.853847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:14.854213 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:14.854214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.854279 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:14.854812 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:14.854816 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:14.856595 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
W20240506 15:41:14.856771 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:14.852406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:38875 (local address 127.17.240.254:43739)
0506 15:41:14.852469 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:14.852471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:14.852625 (+ 154us) server_negotiation.cc:407] Connection header received
0506 15:41:14.852694 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:14.852695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:14.852713 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:14.852760 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:14.853290 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.853773 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:14.854291 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:14.854477 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:14.856701 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.195:38875: BlockingRecv error: recv got EOF from 127.17.240.195:38875 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
May 06 15:41:15 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:15 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.193@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:15.612058 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:15.602681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.193:53957)
0506 15:41:15.602820 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:15.602830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:15.602878 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:15.603089 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:15.603090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:15.603257 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:15.603426 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:15.603428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:15.604128 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:15.604129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:15.604494 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:15.604496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:15.604568 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:15.610000 (+ 5432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:15.610005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:15.611999 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
W20240506 15:41:15.612226 27639 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:15.602787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:53957 (local address 127.17.240.254:43739)
0506 15:41:15.602938 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:15.602939 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:15.602947 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:41:15.602971 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:15.602972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:15.602984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:15.603076 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:15.603512 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:15.604050 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:15.604585 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:15.604714 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:15.612145 (+ 7431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.193:53957: BlockingRecv error: recv got EOF from 127.17.240.193:53957 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
May 06 15:41:15 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
May 06 15:41:15 dist-test-slave-v0k1 krb5kdc[26581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.240.194@KRBTEST.COM for oryx/127.17.240.254@KRBTEST.COM, Server not found in Kerberos database
I20240506 15:41:15.729008 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:15.724805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:43739 (local address 127.17.240.194:48055)
0506 15:41:15.724959 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:15.724970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:15.725040 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:15.725181 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:15.725182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:15.725349 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:15.725506 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:15.725507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:15.726170 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:15.726171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:15.726576 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:15.726578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:15.726645 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:15.727217 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:15.727221 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:15.728925 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.240.254:43739: Server oryx/127.17.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
W20240506 15:41:15.729089 27639 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:15.724907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:48055 (local address 127.17.240.254:43739)
0506 15:41:15.724984 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:15.724986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:15.725025 (+ 39us) server_negotiation.cc:407] Connection header received
0506 15:41:15.725104 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:15.725105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:15.725117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:15.725168 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:15.725592 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:15.726092 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:15.726682 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:15.726840 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:15.729030 (+ 2190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.240.194:48055: BlockingRecv error: recv got EOF from 127.17.240.194:48055 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20240506 15:41:15.795874 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 27164
I20240506 15:41:15.800650 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 27316
I20240506 15:41:15.805749 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 27448
I20240506 15:41:15.810002 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 27094
2024-05-06T15:41:15Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17382 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2024-05-06T15:41:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:41:15Z Disabled control of system clock
I20240506 15:41:15.837379 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:44525
--webserver_interface=127.17.240.254
--webserver_port=0
--builtin_ntp_servers=127.17.240.212:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240506 15:41:15.917322 27644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:15.917531 27644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:15.917552 27644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:15.918944 27644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:41:15.918989 27644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:15.919001 27644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:15.919034 27644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:41:15.919045 27644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:41:15.920428 27644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:44525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:15.920617 27644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:15.920828 27644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:15.925884 27652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:15.926415 27650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:15.927050 27649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:15.927523 27644 server_base.cc:1034] running on GCE node
I20240506 15:41:15.927744 27644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:15.927987 27644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:15.929136 27644 hybrid_clock.cc:648] HybridClock initialized: now 1715010075929118 us; error 38 us; skew 500 ppm
I20240506 15:41:15.930526 27644 webserver.cc:460] Webserver started at http://127.17.240.254:39085/ using document root <none> and password file <none>
I20240506 15:41:15.930752 27644 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:15.930800 27644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:15.930922 27644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:15.932018 27644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "c3496cbf8e3643a3b1845503c8da2490"
format_stamp: "Formatted at 2024-05-06 15:41:15 on dist-test-slave-v0k1"
server_key: "83c8ee835da4c1db8ab569b5b8410d26"
server_key_iv: "f7b6ad170dc78b0b7054df37d81917c4"
server_key_version: "encryptionkey@0"
I20240506 15:41:15.932327 27644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "c3496cbf8e3643a3b1845503c8da2490"
format_stamp: "Formatted at 2024-05-06 15:41:15 on dist-test-slave-v0k1"
server_key: "83c8ee835da4c1db8ab569b5b8410d26"
server_key_iv: "f7b6ad170dc78b0b7054df37d81917c4"
server_key_version: "encryptionkey@0"
I20240506 15:41:15.933526 27644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:41:15.934254 27658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:15.934453 27644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:41:15.934538 27644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "c3496cbf8e3643a3b1845503c8da2490"
format_stamp: "Formatted at 2024-05-06 15:41:15 on dist-test-slave-v0k1"
server_key: "83c8ee835da4c1db8ab569b5b8410d26"
server_key_iv: "f7b6ad170dc78b0b7054df37d81917c4"
server_key_version: "encryptionkey@0"
I20240506 15:41:15.934600 27644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:15.952313 27644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:15.952595 27644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:15.952713 27644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:15.956370 27644 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:44525
I20240506 15:41:15.956403 27710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:44525 every 8 connection(s)
I20240506 15:41:15.956713 27644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:41:15.957307 27711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:41:15.959795 27711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490: Bootstrap starting.
I20240506 15:41:15.960310 27711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490: Neither blocks nor log segments found. Creating new log.
I20240506 15:41:15.960523 27711 log.cc:824] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490: Log is configured to *not* fsync() on all Append() calls
I20240506 15:41:15.961112 27711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490: No bootstrap required, opened a new log
I20240506 15:41:15.961591 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 27644
I20240506 15:41:15.961701 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:41:15.961872 18371 external_mini_cluster.cc:1428] Setting key a9e2c4a9778eebf1a09f439f926b270c
I20240506 15:41:15.962894 27711 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3496cbf8e3643a3b1845503c8da2490" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 44525 } }
I20240506 15:41:15.963065 27711 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:41:15.963097 27711 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3496cbf8e3643a3b1845503c8da2490, State: Initialized, Role: FOLLOWER
I20240506 15:41:15.963234 27711 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3496cbf8e3643a3b1845503c8da2490" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 44525 } }
I20240506 15:41:15.963310 27711 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:41:15.963341 27711 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:41:15.963388 27711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:41:15.964169 27711 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3496cbf8e3643a3b1845503c8da2490" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 44525 } }
I20240506 15:41:15.964290 27711 leader_election.cc:302] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3496cbf8e3643a3b1845503c8da2490; no voters:
I20240506 15:41:15.964439 27711 leader_election.cc:288] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:41:15.964645 27716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:41:15.964679 27711 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:41:15.964802 27716 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [term 1 LEADER]: Becoming Leader. State: Replica: c3496cbf8e3643a3b1845503c8da2490, State: Running, Role: LEADER
I20240506 15:41:15.964936 27716 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3496cbf8e3643a3b1845503c8da2490" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 44525 } }
I20240506 15:41:15.965301 27716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3496cbf8e3643a3b1845503c8da2490. Latest consensus state: current_term: 1 leader_uuid: "c3496cbf8e3643a3b1845503c8da2490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3496cbf8e3643a3b1845503c8da2490" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 44525 } } }
I20240506 15:41:15.965310 27718 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3496cbf8e3643a3b1845503c8da2490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3496cbf8e3643a3b1845503c8da2490" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 44525 } } }
I20240506 15:41:15.965469 27716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [sys.catalog]: This master's current role is: LEADER
I20240506 15:41:15.965476 27718 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490 [sys.catalog]: This master's current role is: LEADER
I20240506 15:41:15.965701 27725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:41:15.966382 27725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:41:15.966974 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:15.962371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.17.240.254:44525)
0506 15:41:15.962609 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:15.962616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:15.962640 (+ 24us) server_negotiation.cc:407] Connection header received
0506 15:41:15.962877 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:15.962885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:15.962946 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:15.963122 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:15.963587 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:15.964795 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:15.965893 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:15.966191 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:15.966328 (+ 137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:15.966336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:15.966336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:41:15.966537 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:15.966571 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:41:15.966579 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:15.966650 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:15.966746 (+ 96us) server_negotiation.cc:299] Negotiation successful
0506 15:41:15.966804 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20240506 15:41:15.967973 27725 catalog_manager.cc:1347] Generated new cluster ID: 3c00727f948747129c2ae22bf4449d65
I20240506 15:41:15.968020 27725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:41:15.975379 27725 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:41:15.975854 27725 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:41:15.984534 27725 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c3496cbf8e3643a3b1845503c8da2490: Generated new TSK 0
I20240506 15:41:15.984726 27725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:41:15.990031 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:44525
--builtin_ntp_servers=127.17.240.212:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240506 15:41:16.070957 27735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:16.071187 27735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:16.071221 27735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:16.072614 27735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:16.072664 27735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:16.072701 27735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:41:16.074131 27735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:16.074380 27735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:16.074632 27735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:16.080091 27740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.080519 27741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.081023 27743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:16.081595 27735 server_base.cc:1034] running on GCE node
I20240506 15:41:16.081799 27735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:16.082062 27735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:16.083220 27735 hybrid_clock.cc:648] HybridClock initialized: now 1715010076083200 us; error 36 us; skew 500 ppm
I20240506 15:41:16.084815 27735 webserver.cc:460] Webserver started at http://127.17.240.193:37973/ using document root <none> and password file <none>
I20240506 15:41:16.085048 27735 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:16.085111 27735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:16.085242 27735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:16.086557 27735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "10fd791f7def4266920f0fbbf893a151"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "2a668fedc5220ada955af7c169ec7dac"
server_key_iv: "cf5c4c2078dc60ec28fd0f0830a0e45f"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.086937 27735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "10fd791f7def4266920f0fbbf893a151"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "2a668fedc5220ada955af7c169ec7dac"
server_key_iv: "cf5c4c2078dc60ec28fd0f0830a0e45f"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.088562 27735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:41:16.089437 27749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.089622 27735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:41:16.089712 27735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "10fd791f7def4266920f0fbbf893a151"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "2a668fedc5220ada955af7c169ec7dac"
server_key_iv: "cf5c4c2078dc60ec28fd0f0830a0e45f"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.089784 27735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:16.127194 27735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:16.127530 27735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:16.127648 27735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:16.127883 27735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:16.128247 27735 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:16.128289 27735 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.128319 27735 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:16.128340 27735 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.134402 27735 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:36471
I20240506 15:41:16.134497 27862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:36471 every 8 connection(s)
I20240506 15:41:16.134763 27735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:41:16.135290 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 27735
I20240506 15:41:16.135404 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:41:16.135614 18371 external_mini_cluster.cc:1428] Setting key 004ca5c7ef0820f0bf70ddeb43c65786
I20240506 15:41:16.136651 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:44525
--builtin_ntp_servers=127.17.240.212:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20240506 15:41:16.140383 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.135516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:53811 (local address 127.17.240.254:44525)
0506 15:41:16.135577 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:16.135579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:16.135864 (+ 285us) server_negotiation.cc:407] Connection header received
0506 15:41:16.136393 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:16.136394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:16.136409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:16.136471 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:16.136995 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.137830 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:16.138716 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.138927 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:16.140083 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:16.140088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:16.140088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:41:16.140095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:16.140113 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:41:16.140117 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:16.140165 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:16.140260 (+ 95us) server_negotiation.cc:299] Negotiation successful
0506 15:41:16.140303 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240506 15:41:16.140574 27865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.135478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:44525 (local address 127.17.240.193:53811)
0506 15:41:16.135732 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:16.135762 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:16.136268 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:16.136518 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:16.136521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:16.136529 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:16.136916 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:16.136920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.137920 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:16.137921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:16.138625 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:16.138628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.138757 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:16.139923 (+ 1166us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:41:16.139924 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:16.139946 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:41:16.139965 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:41:16.140174 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:41:16.140179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:41:16.140182 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:41:16.140258 (+ 76us) client_negotiation.cc:240] Negotiation successful
0506 15:41:16.140338 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":126,"threads_started":1}
I20240506 15:41:16.141103 27863 heartbeater.cc:347] Connected to a master server at 127.17.240.254:44525
I20240506 15:41:16.141223 27863 heartbeater.cc:464] Registering TS with master...
I20240506 15:41:16.141490 27863 heartbeater.cc:511] Master 127.17.240.254:44525 requested a full tablet report, sending...
I20240506 15:41:16.142083 27675 ts_manager.cc:194] Registered new tserver with Master: 10fd791f7def4266920f0fbbf893a151 (127.17.240.193:36471)
I20240506 15:41:16.142781 27675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.193:53811
W20240506 15:41:16.217864 27866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:16.218042 27866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:16.218063 27866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:16.219528 27866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:16.219578 27866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:16.219602 27866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:41:16.221014 27866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:16.221225 27866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:16.221477 27866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:16.226714 27872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.227288 27874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.228153 27871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:16.229022 27866 server_base.cc:1034] running on GCE node
I20240506 15:41:16.229370 27866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:16.229686 27866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:16.230818 27866 hybrid_clock.cc:648] HybridClock initialized: now 1715010076230801 us; error 39 us; skew 500 ppm
I20240506 15:41:16.232223 27866 webserver.cc:460] Webserver started at http://127.17.240.194:33169/ using document root <none> and password file <none>
I20240506 15:41:16.232427 27866 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:16.232475 27866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:16.232600 27866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:16.233597 27866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "fb53eaf34c5f4d17b464f45e8d70b1cd"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "f18ae4e378592687b4f63f7d4c83f11d"
server_key_iv: "755cb2bc5e5fa1d01a56043171d64632"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.233902 27866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "fb53eaf34c5f4d17b464f45e8d70b1cd"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "f18ae4e378592687b4f63f7d4c83f11d"
server_key_iv: "755cb2bc5e5fa1d01a56043171d64632"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.235145 27866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:41:16.235874 27880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.236058 27866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:41:16.236148 27866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "fb53eaf34c5f4d17b464f45e8d70b1cd"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "f18ae4e378592687b4f63f7d4c83f11d"
server_key_iv: "755cb2bc5e5fa1d01a56043171d64632"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.236215 27866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:16.255260 27866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:16.255551 27866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:16.255666 27866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:16.255893 27866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:16.256225 27866 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:16.256261 27866 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.256296 27866 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:16.256322 27866 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.261613 27866 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:43325
I20240506 15:41:16.261663 27993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:43325 every 8 connection(s)
I20240506 15:41:16.261952 27866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:41:16.266300 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.262606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:50849 (local address 127.17.240.254:44525)
0506 15:41:16.262669 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:16.262671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:16.262883 (+ 212us) server_negotiation.cc:407] Connection header received
0506 15:41:16.263387 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:16.263388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:16.263397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:16.263456 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:16.263880 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.264409 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:16.264938 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.265061 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:16.266047 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:16.266051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:16.266052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:41:16.266055 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:16.266069 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:41:16.266073 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:16.266119 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:16.266209 (+ 90us) server_negotiation.cc:299] Negotiation successful
0506 15:41:16.266242 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240506 15:41:16.266400 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.262564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:44525 (local address 127.17.240.194:50849)
0506 15:41:16.262757 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:16.262788 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:16.263300 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:16.263513 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:16.263516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:16.263524 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:16.263822 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:16.263824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.264483 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:16.264484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:16.264863 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:16.264865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.264963 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:16.265904 (+ 941us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:41:16.265905 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:16.265922 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:41:16.265938 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:41:16.266131 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:41:16.266134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:41:16.266138 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:41:16.266204 (+ 66us) client_negotiation.cc:240] Negotiation successful
0506 15:41:16.266258 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20240506 15:41:16.266839 27994 heartbeater.cc:347] Connected to a master server at 127.17.240.254:44525
I20240506 15:41:16.266927 27994 heartbeater.cc:464] Registering TS with master...
I20240506 15:41:16.267153 27994 heartbeater.cc:511] Master 127.17.240.254:44525 requested a full tablet report, sending...
I20240506 15:41:16.267503 27675 ts_manager.cc:194] Registered new tserver with Master: fb53eaf34c5f4d17b464f45e8d70b1cd (127.17.240.194:43325)
I20240506 15:41:16.267872 27675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.194:50849
I20240506 15:41:16.272125 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 27866
I20240506 15:41:16.272231 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:41:16.272413 18371 external_mini_cluster.cc:1428] Setting key dba0cec952730cad9edc155766a9db37
I20240506 15:41:16.273365 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:44525
--builtin_ntp_servers=127.17.240.212:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240506 15:41:16.352137 27997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:16.352309 27997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:16.352341 27997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:16.353621 27997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:16.353664 27997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:16.353698 27997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:41:16.355074 27997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:16.355291 27997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:16.355515 27997 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:16.360378 28002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.360971 28003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.361461 28005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:16.362074 27997 server_base.cc:1034] running on GCE node
I20240506 15:41:16.362278 27997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:16.362499 27997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:16.363637 27997 hybrid_clock.cc:648] HybridClock initialized: now 1715010076363611 us; error 41 us; skew 500 ppm
I20240506 15:41:16.364955 27997 webserver.cc:460] Webserver started at http://127.17.240.195:45691/ using document root <none> and password file <none>
I20240506 15:41:16.365180 27997 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:16.365235 27997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:16.365360 27997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:16.366403 27997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "ddc9d479399847e2a669654cd35b7d15"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "b4725d658ce7ed7b9fc72dc2fe53e53b"
server_key_iv: "3b41436fb5a4f337e09c3f29ee80af84"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.366704 27997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "ddc9d479399847e2a669654cd35b7d15"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "b4725d658ce7ed7b9fc72dc2fe53e53b"
server_key_iv: "3b41436fb5a4f337e09c3f29ee80af84"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.367892 27997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:41:16.368561 28011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.368748 27997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:41:16.368814 27997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "ddc9d479399847e2a669654cd35b7d15"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "b4725d658ce7ed7b9fc72dc2fe53e53b"
server_key_iv: "3b41436fb5a4f337e09c3f29ee80af84"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.368877 27997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:16.401638 27997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:16.401933 27997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:16.402047 27997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:16.402292 27997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:16.402604 27997 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:16.402639 27997 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.402662 27997 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:16.402674 27997 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.408159 27997 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:40335
I20240506 15:41:16.408217 28124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:40335 every 8 connection(s)
I20240506 15:41:16.408494 27997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:41:16.412773 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.409171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:55281 (local address 127.17.240.254:44525)
0506 15:41:16.409216 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:16.409218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:16.409443 (+ 225us) server_negotiation.cc:407] Connection header received
0506 15:41:16.409906 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:16.409907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:16.409919 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:16.409981 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:16.410406 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.410908 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:16.411458 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.411579 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:16.412515 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:16.412520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:16.412520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:41:16.412524 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:16.412553 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:41:16.412556 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:16.412600 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:16.412681 (+ 81us) server_negotiation.cc:299] Negotiation successful
0506 15:41:16.412713 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240506 15:41:16.412868 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.409157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:44525 (local address 127.17.240.195:55281)
0506 15:41:16.409343 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:16.409372 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:16.409822 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:16.410036 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:16.410039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:16.410047 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:16.410346 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:16.410349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.410984 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:16.410985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:16.411370 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:16.411372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:16.411488 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:16.412401 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:41:16.412402 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:16.412419 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:41:16.412436 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:41:16.412608 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:41:16.412610 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:41:16.412613 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:41:16.412678 (+ 65us) client_negotiation.cc:240] Negotiation successful
0506 15:41:16.412731 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20240506 15:41:16.413307 28125 heartbeater.cc:347] Connected to a master server at 127.17.240.254:44525
I20240506 15:41:16.413390 28125 heartbeater.cc:464] Registering TS with master...
I20240506 15:41:16.413565 28125 heartbeater.cc:511] Master 127.17.240.254:44525 requested a full tablet report, sending...
I20240506 15:41:16.413909 27675 ts_manager.cc:194] Registered new tserver with Master: ddc9d479399847e2a669654cd35b7d15 (127.17.240.195:40335)
I20240506 15:41:16.414235 27675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.195:55281
I20240506 15:41:16.418267 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 27997
I20240506 15:41:16.418352 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:41:16.418543 18371 external_mini_cluster.cc:1428] Setting key 9e58774fa6cdc751b5ed07e8d479cf11
I20240506 15:41:16.419463 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240506 15:41:16.421145 27715 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:16.420657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.17.240.254:44525)
0506 15:41:16.420716 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:16.420718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:16.420823 (+ 105us) server_negotiation.cc:407] Connection header received
0506 15:41:16.420888 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:16.420889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:16.420901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:16.420949 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:16.421085 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36896: BlockingRecv error: recv got EOF from 127.0.0.1:36896 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20240506 15:41:16.421202 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 27735
I20240506 15:41:16.425910 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 27866
I20240506 15:41:16.430416 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 27997
I20240506 15:41:16.434628 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 27644
2024-05-06T15:41:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (623 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2024-05-06T15:41:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:41:16Z Disabled control of system clock
I20240506 15:41:16.460786 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:46489
--webserver_interface=127.17.240.254
--webserver_port=0
--builtin_ntp_servers=127.17.240.212:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240506 15:41:16.540366 28136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:16.540547 28136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:16.540567 28136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:16.541852 28136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:41:16.541898 28136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:16.541909 28136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:16.541922 28136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:41:16.541949 28136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:41:16.543380 28136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:46489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:16.543587 28136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:16.543823 28136 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:16.548323 28144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.548751 28142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.549190 28141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:16.549752 28136 server_base.cc:1034] running on GCE node
I20240506 15:41:16.549965 28136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:16.550177 28136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:16.551301 28136 hybrid_clock.cc:648] HybridClock initialized: now 1715010076551279 us; error 37 us; skew 500 ppm
I20240506 15:41:16.552683 28136 webserver.cc:460] Webserver started at http://127.17.240.254:34561/ using document root <none> and password file <none>
I20240506 15:41:16.552884 28136 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:16.552932 28136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:16.553045 28136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:16.554077 28136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "1498f0ad70534ebaa22797ae609137a2"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "39ccb2ae4737e7ef41a6c20629650ea1"
server_key_iv: "b60cfa4790ef9f84db0bd81e4f4c6e8a"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.554380 28136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "1498f0ad70534ebaa22797ae609137a2"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "39ccb2ae4737e7ef41a6c20629650ea1"
server_key_iv: "b60cfa4790ef9f84db0bd81e4f4c6e8a"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.555663 28136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:41:16.556394 28150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.556591 28136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:41:16.556669 28136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "1498f0ad70534ebaa22797ae609137a2"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "39ccb2ae4737e7ef41a6c20629650ea1"
server_key_iv: "b60cfa4790ef9f84db0bd81e4f4c6e8a"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.556732 28136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:16.560256 28136 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:16.560492 28136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:16.560602 28136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:16.564182 28136 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:46489
I20240506 15:41:16.564365 28202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:46489 every 8 connection(s)
I20240506 15:41:16.564555 28136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:41:16.565174 28203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:41:16.567772 28203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2: Bootstrap starting.
I20240506 15:41:16.568293 28203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2: Neither blocks nor log segments found. Creating new log.
I20240506 15:41:16.568568 28203 log.cc:824] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2: Log is configured to *not* fsync() on all Append() calls
I20240506 15:41:16.569234 28203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2: No bootstrap required, opened a new log
I20240506 15:41:16.570820 28203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1498f0ad70534ebaa22797ae609137a2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46489 } }
I20240506 15:41:16.570950 28203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:41:16.570967 28203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1498f0ad70534ebaa22797ae609137a2, State: Initialized, Role: FOLLOWER
I20240506 15:41:16.571115 28203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1498f0ad70534ebaa22797ae609137a2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46489 } }
I20240506 15:41:16.571190 28203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:41:16.571223 28203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:41:16.571282 28203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:41:16.571861 28203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1498f0ad70534ebaa22797ae609137a2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46489 } }
I20240506 15:41:16.571951 28203 leader_election.cc:302] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1498f0ad70534ebaa22797ae609137a2; no voters:
I20240506 15:41:16.572099 28203 leader_election.cc:288] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:41:16.572160 28206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:41:16.572332 28203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:41:16.572336 28206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [term 1 LEADER]: Becoming Leader. State: Replica: 1498f0ad70534ebaa22797ae609137a2, State: Running, Role: LEADER
I20240506 15:41:16.572441 28206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1498f0ad70534ebaa22797ae609137a2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46489 } }
I20240506 15:41:16.572798 28207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1498f0ad70534ebaa22797ae609137a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1498f0ad70534ebaa22797ae609137a2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46489 } } }
I20240506 15:41:16.572937 28207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [sys.catalog]: This master's current role is: LEADER
I20240506 15:41:16.573216 28215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:41:16.572815 28208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1498f0ad70534ebaa22797ae609137a2. Latest consensus state: current_term: 1 leader_uuid: "1498f0ad70534ebaa22797ae609137a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1498f0ad70534ebaa22797ae609137a2" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46489 } } }
I20240506 15:41:16.573379 28208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2 [sys.catalog]: This master's current role is: LEADER
I20240506 15:41:16.573722 28215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:41:16.574661 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 28136
I20240506 15:41:16.574751 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:41:16.574921 18371 external_mini_cluster.cc:1428] Setting key 13e698846d1dcdc56b8ce82c034f248b
I20240506 15:41:16.575210 28215 catalog_manager.cc:1347] Generated new cluster ID: ee2246696527493b9df0ab69818bb602
I20240506 15:41:16.575269 28215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:41:16.577214 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.575449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.17.240.254:46489)
0506 15:41:16.575645 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:16.575652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:16.575853 (+ 201us) server_negotiation.cc:407] Connection header received
0506 15:41:16.575983 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:16.575989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:16.576025 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:16.576147 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:16.576838 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:16.576840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:16.576840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:41:16.576844 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:16.576867 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:41:16.576873 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:16.576950 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:16.577020 (+ 70us) server_negotiation.cc:299] Negotiation successful
0506 15:41:16.577060 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20240506 15:41:16.587764 28215 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:41:16.587924 28215 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:41:16.594297 28215 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1498f0ad70534ebaa22797ae609137a2: Generated new TSK 0
I20240506 15:41:16.594489 28215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:41:16.598176 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46489
--builtin_ntp_servers=127.17.240.212:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240506 15:41:16.701597 28227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:16.701802 28227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:16.701836 28227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:16.703222 28227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:16.703271 28227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:16.703310 28227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:41:16.704881 28227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:16.705119 28227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:16.705353 28227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:16.710687 28232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.711139 28233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.711701 28235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:16.712296 28227 server_base.cc:1034] running on GCE node
I20240506 15:41:16.712476 28227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:16.712697 28227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:16.713850 28227 hybrid_clock.cc:648] HybridClock initialized: now 1715010076713835 us; error 34 us; skew 500 ppm
I20240506 15:41:16.715195 28227 webserver.cc:460] Webserver started at http://127.17.240.193:37705/ using document root <none> and password file <none>
I20240506 15:41:16.715392 28227 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:16.715436 28227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:16.715570 28227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:16.716598 28227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "e94bc421e0f04f4eade821195f24daf4"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "4161b5c5e3e910385627385364c6cfcc"
server_key_iv: "82f0de425b09afe04539fa21666369d5"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.716900 28227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "e94bc421e0f04f4eade821195f24daf4"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "4161b5c5e3e910385627385364c6cfcc"
server_key_iv: "82f0de425b09afe04539fa21666369d5"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.718062 28227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:41:16.718732 28241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.718880 28227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:41:16.718956 28227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "e94bc421e0f04f4eade821195f24daf4"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "4161b5c5e3e910385627385364c6cfcc"
server_key_iv: "82f0de425b09afe04539fa21666369d5"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.719048 28227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:16.722687 28227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:16.722916 28227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:16.723037 28227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:16.723229 28227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:16.723523 28227 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:16.723556 28227 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.723589 28227 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:16.723627 28227 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.728756 28227 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:44319
I20240506 15:41:16.728847 28354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:44319 every 8 connection(s)
I20240506 15:41:16.729110 28227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:41:16.731712 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.729856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:47135 (local address 127.17.240.254:46489)
0506 15:41:16.729907 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:16.729909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:16.730098 (+ 189us) server_negotiation.cc:407] Connection header received
0506 15:41:16.730545 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:16.730547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:16.730553 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:16.730640 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:16.731459 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:16.731461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:16.731461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:41:16.731467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:16.731488 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:41:16.731492 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:16.731573 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:16.731621 (+ 48us) server_negotiation.cc:299] Negotiation successful
0506 15:41:16.731637 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240506 15:41:16.731820 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.729807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:46489 (local address 127.17.240.193:47135)
0506 15:41:16.729986 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:16.730014 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:16.730445 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:16.730681 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:16.730684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:16.730692 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:16.731307 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:41:16.731308 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:16.731327 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:41:16.731350 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:41:16.731566 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:41:16.731568 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:41:16.731570 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:41:16.731639 (+ 69us) client_negotiation.cc:240] Negotiation successful
0506 15:41:16.731676 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20240506 15:41:16.732802 28355 heartbeater.cc:347] Connected to a master server at 127.17.240.254:46489
I20240506 15:41:16.732913 28355 heartbeater.cc:464] Registering TS with master...
I20240506 15:41:16.733062 28355 heartbeater.cc:511] Master 127.17.240.254:46489 requested a full tablet report, sending...
I20240506 15:41:16.733543 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 28227
I20240506 15:41:16.733650 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:41:16.734006 18371 external_mini_cluster.cc:1428] Setting key 6b4b9fefc9c33a127c0d12794eece5e6
I20240506 15:41:16.734572 28166 ts_manager.cc:194] Registered new tserver with Master: e94bc421e0f04f4eade821195f24daf4 (127.17.240.193:44319)
I20240506 15:41:16.735157 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46489
--builtin_ntp_servers=127.17.240.212:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240506 15:41:16.815563 28358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:16.815752 28358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:16.815778 28358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:16.817126 28358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:16.817171 28358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:16.817196 28358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:41:16.818639 28358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:16.818851 28358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:16.819124 28358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:16.824294 28366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.824730 28364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.825311 28363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:16.825976 28358 server_base.cc:1034] running on GCE node
I20240506 15:41:16.826191 28358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:16.826421 28358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:16.827564 28358 hybrid_clock.cc:648] HybridClock initialized: now 1715010076827566 us; error 40 us; skew 500 ppm
I20240506 15:41:16.828934 28358 webserver.cc:460] Webserver started at http://127.17.240.194:33597/ using document root <none> and password file <none>
I20240506 15:41:16.829138 28358 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:16.829185 28358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:16.829294 28358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:16.830324 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "4123f1a58f7e4a5cbab4d74378504b29"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "0d5b446a42868ec5ce98371953361d82"
server_key_iv: "363a69b9bc39da3e616e64f680d19e3e"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.830641 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "4123f1a58f7e4a5cbab4d74378504b29"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "0d5b446a42868ec5ce98371953361d82"
server_key_iv: "363a69b9bc39da3e616e64f680d19e3e"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.831965 28358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:41:16.832790 28372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.833021 28358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:41:16.833120 28358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "4123f1a58f7e4a5cbab4d74378504b29"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "0d5b446a42868ec5ce98371953361d82"
server_key_iv: "363a69b9bc39da3e616e64f680d19e3e"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.833184 28358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:16.836946 28358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:16.837172 28358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:16.837275 28358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:16.837438 28358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:16.837728 28358 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:16.837761 28358 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.837795 28358 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:16.837821 28358 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.843294 28358 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:39107
I20240506 15:41:16.843351 28485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:39107 every 8 connection(s)
I20240506 15:41:16.843648 28358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:41:16.846217 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.844440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:50233 (local address 127.17.240.254:46489)
0506 15:41:16.844502 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:16.844504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:16.844632 (+ 128us) server_negotiation.cc:407] Connection header received
0506 15:41:16.845090 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:16.845091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:16.845097 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:16.845164 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:16.845973 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:16.845975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:16.845975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:41:16.845979 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:16.845995 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:41:16.845999 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:16.846058 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:16.846142 (+ 84us) server_negotiation.cc:299] Negotiation successful
0506 15:41:16.846157 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240506 15:41:16.846377 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.844316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:46489 (local address 127.17.240.194:50233)
0506 15:41:16.844505 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:16.844534 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:16.844961 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:16.845224 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:16.845228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:16.845235 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:16.845814 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:41:16.845814 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:16.845836 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:41:16.845860 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:41:16.846065 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:41:16.846068 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:41:16.846070 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:41:16.846153 (+ 83us) client_negotiation.cc:240] Negotiation successful
0506 15:41:16.846195 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20240506 15:41:16.847193 28486 heartbeater.cc:347] Connected to a master server at 127.17.240.254:46489
I20240506 15:41:16.847282 28486 heartbeater.cc:464] Registering TS with master...
I20240506 15:41:16.847419 28486 heartbeater.cc:511] Master 127.17.240.254:46489 requested a full tablet report, sending...
I20240506 15:41:16.847839 28166 ts_manager.cc:194] Registered new tserver with Master: 4123f1a58f7e4a5cbab4d74378504b29 (127.17.240.194:39107)
I20240506 15:41:16.849817 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 28358
I20240506 15:41:16.849922 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:41:16.850069 18371 external_mini_cluster.cc:1428] Setting key 27716e4068aca4efe4b21d33791c37a8
I20240506 15:41:16.851258 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46489
--builtin_ntp_servers=127.17.240.212:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240506 15:41:16.931180 28489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:16.931434 28489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:16.931473 28489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:16.933001 28489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:16.933054 28489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:16.933081 28489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:41:16.934499 28489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:16.934686 28489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:16.934890 28489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:16.939805 28497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.940325 28495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:16.940798 28494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:16.941395 28489 server_base.cc:1034] running on GCE node
I20240506 15:41:16.941591 28489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:16.941848 28489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:16.942977 28489 hybrid_clock.cc:648] HybridClock initialized: now 1715010076942956 us; error 44 us; skew 500 ppm
I20240506 15:41:16.944288 28489 webserver.cc:460] Webserver started at http://127.17.240.195:36703/ using document root <none> and password file <none>
I20240506 15:41:16.944489 28489 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:16.944531 28489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:16.944661 28489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:16.945680 28489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "627fbe1aed624cb0a5ab92edcb750827"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "1db8a080fd5ffe42ab8b7bd80fbc379d"
server_key_iv: "417833547be5e72a22a7d6121fd9ecc2"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.945998 28489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "627fbe1aed624cb0a5ab92edcb750827"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "1db8a080fd5ffe42ab8b7bd80fbc379d"
server_key_iv: "417833547be5e72a22a7d6121fd9ecc2"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.947197 28489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:41:16.947858 28503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.948040 28489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:41:16.948117 28489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "627fbe1aed624cb0a5ab92edcb750827"
format_stamp: "Formatted at 2024-05-06 15:41:16 on dist-test-slave-v0k1"
server_key: "1db8a080fd5ffe42ab8b7bd80fbc379d"
server_key_iv: "417833547be5e72a22a7d6121fd9ecc2"
server_key_version: "encryptionkey@0"
I20240506 15:41:16.948190 28489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:16.951750 28489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:16.951967 28489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:16.952068 28489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:16.952255 28489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:16.952529 28489 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:16.952561 28489 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.952595 28489 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:16.952625 28489 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:16.957947 28489 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:35615
I20240506 15:41:16.957983 28616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:35615 every 8 connection(s)
I20240506 15:41:16.958262 28489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:41:16.960741 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.959052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:43637 (local address 127.17.240.254:46489)
0506 15:41:16.959240 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:16.959242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:16.959306 (+ 64us) server_negotiation.cc:407] Connection header received
0506 15:41:16.959777 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:16.959779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:16.959784 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:16.959847 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:16.960534 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:16.960535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:16.960535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:41:16.960539 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:16.960555 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:41:16.960559 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:16.960613 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:16.960677 (+ 64us) server_negotiation.cc:299] Negotiation successful
0506 15:41:16.960688 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20240506 15:41:16.960888 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:16.958987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:46489 (local address 127.17.240.195:43637)
0506 15:41:16.959211 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:16.959233 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:16.959668 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:16.959892 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:16.959895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:16.959903 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:16.960414 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:41:16.960414 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:16.960431 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:41:16.960452 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:41:16.960612 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:41:16.960615 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:41:16.960617 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:41:16.960690 (+ 73us) client_negotiation.cc:240] Negotiation successful
0506 15:41:16.960726 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":94,"threads_started":1}
I20240506 15:41:16.961500 28617 heartbeater.cc:347] Connected to a master server at 127.17.240.254:46489
I20240506 15:41:16.961582 28617 heartbeater.cc:464] Registering TS with master...
I20240506 15:41:16.961737 28617 heartbeater.cc:511] Master 127.17.240.254:46489 requested a full tablet report, sending...
I20240506 15:41:16.962131 28166 ts_manager.cc:194] Registered new tserver with Master: 627fbe1aed624cb0a5ab92edcb750827 (127.17.240.195:35615)
I20240506 15:41:16.965649 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 28489
I20240506 15:41:16.965741 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:41:16.965922 18371 external_mini_cluster.cc:1428] Setting key 37928aaad775d46881a151f225961db7
I20240506 15:41:16.967209 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240506 15:41:16.969045 28226 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:41:16.968562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.17.240.254:46489)
0506 15:41:16.968633 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:16.968634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:16.968757 (+ 123us) server_negotiation.cc:407] Connection header received
0506 15:41:16.968785 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:16.968786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:16.968791 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:16.968848 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:16.968967 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57668: BlockingRecv error: recv got EOF from 127.0.0.1:57668 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20240506 15:41:16.969182 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 28227
I20240506 15:41:16.974336 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 28358
I20240506 15:41:16.978994 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 28489
I20240506 15:41:16.983644 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 28136
2024-05-06T15:41:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (548 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:41:17 dist-test-slave-v0k1 krb5kdc[28627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:41:17 dist-test-slave-v0k1 krb5kdc[28627](info): set up 2 sockets
May 06 15:41:17 dist-test-slave-v0k1 krb5kdc[28627](info): commencing operation
krb5kdc: starting...
W20240506 15:41:17.947362 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[28627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:41:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:41:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:41:18.063905 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38557
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:41:18.145996 28640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:18.146200 28640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:18.146224 28640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:18.147573 28640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:41:18.147624 28640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:18.147639 28640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:18.147658 28640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:41:18.147680 28640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:41:18.149341 28640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:38557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:18.149559 28640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:18.149745 28640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:18.155392 28646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:18.155771 28645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:18.156387 28648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:18.157002 28640 server_base.cc:1034] running on GCE node
I20240506 15:41:18.157284 28640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:18.157552 28640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:18.158715 28640 hybrid_clock.cc:648] HybridClock initialized: now 1715010078158695 us; error 35 us; skew 500 ppm
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[28627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010078, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:41:18.161357 28640 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:41:18.161809 28640 webserver.cc:460] Webserver started at http://127.17.240.254:34599/ using document root <none> and password file <none>
I20240506 15:41:18.162009 28640 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:18.162055 28640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:18.162168 28640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:18.163214 28640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "4273d8a5c8db41368decc5043fab74d7"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "71f80b7d9cd478a750cc92b4095c2a54"
server_key_iv: "235959ae57807125416cb8def5cca84d"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.163518 28640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "4273d8a5c8db41368decc5043fab74d7"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "71f80b7d9cd478a750cc92b4095c2a54"
server_key_iv: "235959ae57807125416cb8def5cca84d"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.164685 28640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:41:18.165313 28655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.165465 28640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:41:18.165544 28640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "4273d8a5c8db41368decc5043fab74d7"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "71f80b7d9cd478a750cc92b4095c2a54"
server_key_iv: "235959ae57807125416cb8def5cca84d"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.165613 28640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:18.193096 28640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:18.195042 28640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:18.195168 28640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:18.199095 28640 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:38557
I20240506 15:41:18.199128 28707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:38557 every 8 connection(s)
I20240506 15:41:18.199468 28640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:41:18.200299 28708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:41:18.202379 28708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7: Bootstrap starting.
I20240506 15:41:18.202960 28708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7: Neither blocks nor log segments found. Creating new log.
I20240506 15:41:18.203230 28708 log.cc:824] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7: Log is configured to *not* fsync() on all Append() calls
I20240506 15:41:18.203866 28708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7: No bootstrap required, opened a new log
I20240506 15:41:18.205267 28708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4273d8a5c8db41368decc5043fab74d7" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38557 } }
I20240506 15:41:18.205385 28708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:41:18.205402 28708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4273d8a5c8db41368decc5043fab74d7, State: Initialized, Role: FOLLOWER
I20240506 15:41:18.205495 28708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4273d8a5c8db41368decc5043fab74d7" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38557 } }
I20240506 15:41:18.205561 28708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:41:18.205587 28708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:41:18.205631 28708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:41:18.206141 28708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4273d8a5c8db41368decc5043fab74d7" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38557 } }
I20240506 15:41:18.206224 28708 leader_election.cc:302] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4273d8a5c8db41368decc5043fab74d7; no voters:
I20240506 15:41:18.206357 28708 leader_election.cc:288] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:41:18.206413 28711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:41:18.206578 28708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:41:18.206607 28711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [term 1 LEADER]: Becoming Leader. State: Replica: 4273d8a5c8db41368decc5043fab74d7, State: Running, Role: LEADER
I20240506 15:41:18.206724 28711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4273d8a5c8db41368decc5043fab74d7" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38557 } }
I20240506 15:41:18.207105 28712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4273d8a5c8db41368decc5043fab74d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4273d8a5c8db41368decc5043fab74d7" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38557 } } }
I20240506 15:41:18.207131 28713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4273d8a5c8db41368decc5043fab74d7. Latest consensus state: current_term: 1 leader_uuid: "4273d8a5c8db41368decc5043fab74d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4273d8a5c8db41368decc5043fab74d7" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38557 } } }
I20240506 15:41:18.207258 28712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [sys.catalog]: This master's current role is: LEADER
I20240506 15:41:18.207331 28713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7 [sys.catalog]: This master's current role is: LEADER
I20240506 15:41:18.207672 28718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:41:18.208156 28718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:41:18.208669 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 28640
I20240506 15:41:18.208773 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:41:18.208966 18371 external_mini_cluster.cc:1428] Setting key 5bd22157b6fe528d7ae6b89e2376007e
I20240506 15:41:18.209519 28718 catalog_manager.cc:1347] Generated new cluster ID: 1554e9a2efd44b0da5b04d26245735f7
I20240506 15:41:18.209560 28718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[28627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:41:18.221004 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.209390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.17.240.254:38557)
0506 15:41:18.211309 (+ 1919us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.211320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.211341 (+ 21us) server_negotiation.cc:407] Connection header received
0506 15:41:18.211448 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.211453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.211497 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.211612 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:18.212576 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.213714 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.214490 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.214705 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.217455 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:18.217462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:18.217462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:41:18.217482 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:18.219603 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:18.219947 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:18.219949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:18.219950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:18.220011 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:18.220172 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:18.220173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:18.220173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:18.220430 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:18.220581 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.220660 (+ 79us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.220788 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1883,"thread_start_us":60,"threads_started":1}
I20240506 15:41:18.222358 28718 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:41:18.222950 28718 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:41:18.238318 28718 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7: Generated new TSK 0
I20240506 15:41:18.238530 28718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:41:18.266249 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38557
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:41:18.346555 28734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:18.346765 28734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:18.346796 28734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:18.348146 28734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:18.348193 28734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:18.348233 28734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:41:18.349680 28734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:18.349905 28734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:18.350085 28734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:18.354668 28740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:18.355219 28739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:18.355706 28742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:18.356243 28734 server_base.cc:1034] running on GCE node
I20240506 15:41:18.356416 28734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:18.356631 28734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:18.357776 28734 hybrid_clock.cc:648] HybridClock initialized: now 1715010078357758 us; error 38 us; skew 500 ppm
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[28627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010078, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:41:18.360373 28734 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:41:18.360913 28734 webserver.cc:460] Webserver started at http://127.17.240.193:44845/ using document root <none> and password file <none>
I20240506 15:41:18.361135 28734 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:18.361200 28734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:18.361311 28734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:18.362519 28734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "8b59e762d84872d23258cd8019f887c5"
server_key_iv: "f76a2c9626215ddd2723cdc7f9e8c603"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.362848 28734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "8b59e762d84872d23258cd8019f887c5"
server_key_iv: "f76a2c9626215ddd2723cdc7f9e8c603"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.364301 28734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:41:18.365092 28749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.365257 28734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.365335 28734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "8b59e762d84872d23258cd8019f887c5"
server_key_iv: "f76a2c9626215ddd2723cdc7f9e8c603"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.365401 28734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:18.381356 28734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:18.383698 28734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:18.383826 28734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:18.384073 28734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:18.384370 28734 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:18.384403 28734 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.384438 28734 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:18.384465 28734 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.390084 28734 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:42175
I20240506 15:41:18.390146 28862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:42175 every 8 connection(s)
I20240506 15:41:18.390431 28734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:41:18.391214 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 28734
I20240506 15:41:18.391330 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:41:18.391547 18371 external_mini_cluster.cc:1428] Setting key a173cd48f26258f81872e7aa33d2adef
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[28627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010078, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
I20240506 15:41:18.399919 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.391131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:39245 (local address 127.17.240.254:38557)
0506 15:41:18.391188 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.391191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.391436 (+ 245us) server_negotiation.cc:407] Connection header received
0506 15:41:18.392140 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.392150 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.392169 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.392246 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:18.393329 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.393983 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.394766 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.394873 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.397138 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:18.397142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:18.397142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:41:18.397149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:18.398603 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:18.398989 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:18.398991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:18.398991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:18.399052 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:18.399296 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:18.399297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:18.399297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:18.399389 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:18.399458 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.399717 (+ 259us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.399820 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:41:18.400023 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.391136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38557 (local address 127.17.240.193:39245)
0506 15:41:18.391305 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:18.391338 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:18.392007 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:18.392313 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:18.392318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:18.392655 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:18.393223 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.393228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.394087 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.394088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:18.394684 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.394686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.394829 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.395417 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:18.395443 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:18.397035 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:41:18.398688 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:41:18.398695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:41:18.398698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:41:18.398920 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:41:18.399110 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:41:18.399111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:41:18.399112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:41:18.399176 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:41:18.399457 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:41:18.399463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:41:18.399590 (+ 127us) client_negotiation.cc:769] Sending connection context
0506 15:41:18.399690 (+ 100us) client_negotiation.cc:240] Negotiation successful
0506 15:41:18.399834 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20240506 15:41:18.400501 28863 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38557
I20240506 15:41:18.400607 28863 heartbeater.cc:464] Registering TS with master...
I20240506 15:41:18.400810 28863 heartbeater.cc:511] Master 127.17.240.254:38557 requested a full tablet report, sending...
I20240506 15:41:18.401360 28672 ts_manager.cc:194] Registered new tserver with Master: 68e9e76d5ec24d1c9c4466d7446f5b7f (127.17.240.193:42175)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
I20240506 15:41:18.402112 28672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:39245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:41:18.417941 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38557
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:41:18.497845 28868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:18.498052 28868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:18.498076 28868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:18.499485 28868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:18.499534 28868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:18.499562 28868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:41:18.501084 28868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:18.501292 28868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:18.501495 28868 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:18.506357 28874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:18.506738 28873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:18.507220 28876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:18.507804 28868 server_base.cc:1034] running on GCE node
I20240506 15:41:18.507997 28868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:18.508188 28868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:18.509327 28868 hybrid_clock.cc:648] HybridClock initialized: now 1715010078509316 us; error 32 us; skew 500 ppm
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[28627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010078, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:41:18.511906 28868 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:41:18.512346 28868 webserver.cc:460] Webserver started at http://127.17.240.194:43267/ using document root <none> and password file <none>
I20240506 15:41:18.512539 28868 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:18.512579 28868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:18.512688 28868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:18.513731 28868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "f01691e975f84377ab86a51900758dc4"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "307fc7671756e416561e7314e808ba57"
server_key_iv: "e7bb791906f7f787336c7433430fcd2e"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.514055 28868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "f01691e975f84377ab86a51900758dc4"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "307fc7671756e416561e7314e808ba57"
server_key_iv: "e7bb791906f7f787336c7433430fcd2e"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.515332 28868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:41:18.516033 28883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.516239 28868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:41:18.516325 28868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "f01691e975f84377ab86a51900758dc4"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "307fc7671756e416561e7314e808ba57"
server_key_iv: "e7bb791906f7f787336c7433430fcd2e"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.516403 28868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:18.524416 28868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:18.526422 28868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:18.526527 28868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:18.526746 28868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:18.527093 28868 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:18.527132 28868 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.527154 28868 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:18.527168 28868 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.532570 28868 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:43731
I20240506 15:41:18.532606 28996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:43731 every 8 connection(s)
I20240506 15:41:18.532888 28868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[28627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010078, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:41:18.541061 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.533569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:52337 (local address 127.17.240.254:38557)
0506 15:41:18.533648 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.533651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.533773 (+ 122us) server_negotiation.cc:407] Connection header received
0506 15:41:18.534263 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.534264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.534276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.534344 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:18.535433 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.535912 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.536439 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.536620 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.538518 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:18.538522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:18.538523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:41:18.538529 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:18.539872 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:18.540259 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:18.540261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:18.540261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:18.540286 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:18.540459 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:18.540460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:18.540461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:18.540565 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:18.540626 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.540866 (+ 240us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.540963 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240506 15:41:18.541183 28999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.533494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38557 (local address 127.17.240.194:52337)
0506 15:41:18.533649 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:18.533681 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:18.534166 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:18.534404 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:18.534407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:18.534703 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:18.535332 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.535337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.535992 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.535993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:18.536355 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.536357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.536485 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.537083 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:18.537097 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:18.538424 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:41:18.539955 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:41:18.539961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:41:18.539964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:41:18.540188 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:41:18.540343 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:41:18.540344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:41:18.540344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:41:18.540404 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:41:18.540620 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:41:18.540625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:41:18.540743 (+ 118us) client_negotiation.cc:769] Sending connection context
0506 15:41:18.540842 (+ 99us) client_negotiation.cc:240] Negotiation successful
0506 15:41:18.540979 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20240506 15:41:18.541671 28997 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38557
I20240506 15:41:18.541764 28997 heartbeater.cc:464] Registering TS with master...
I20240506 15:41:18.541934 28997 heartbeater.cc:511] Master 127.17.240.254:38557 requested a full tablet report, sending...
I20240506 15:41:18.542363 28672 ts_manager.cc:194] Registered new tserver with Master: f01691e975f84377ab86a51900758dc4 (127.17.240.194:43731)
I20240506 15:41:18.542557 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 28868
I20240506 15:41:18.542670 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:41:18.542840 18371 external_mini_cluster.cc:1428] Setting key 1a55ed4d3d7cce3c7c34593ec222907d
I20240506 15:41:18.542943 28672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:52337
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:41:18.567920 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38557
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:41:18.647552 29002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:41:18.647802 29002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:41:18.647847 29002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:41:18.650233 29002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:41:18.650305 29002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:41:18.650350 29002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:41:18.652580 29002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:41:18.652842 29002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:41:18.653041 29002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:41:18.657730 29007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:18.658242 29010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:41:18.658737 29008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:41:18.659266 29002 server_base.cc:1034] running on GCE node
I20240506 15:41:18.659457 29002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:41:18.659682 29002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:41:18.660854 29002 hybrid_clock.cc:648] HybridClock initialized: now 1715010078660841 us; error 24 us; skew 500 ppm
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[28627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010078, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:41:18.663422 29002 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:41:18.663856 29002 webserver.cc:460] Webserver started at http://127.17.240.195:39659/ using document root <none> and password file <none>
I20240506 15:41:18.664032 29002 fs_manager.cc:359] Metadata directory not provided
I20240506 15:41:18.664077 29002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:41:18.664186 29002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:41:18.665278 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "99a62b226d1141cd89d1c925befcc9ca"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "c9a5bc51e818e75a473d63fdd93d7cb9"
server_key_iv: "bff80b59dbe3cfdb89898db68406e435"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.665642 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "99a62b226d1141cd89d1c925befcc9ca"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "c9a5bc51e818e75a473d63fdd93d7cb9"
server_key_iv: "bff80b59dbe3cfdb89898db68406e435"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.666836 29002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:41:18.667670 29017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.667833 29002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:41:18.667912 29002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "99a62b226d1141cd89d1c925befcc9ca"
format_stamp: "Formatted at 2024-05-06 15:41:18 on dist-test-slave-v0k1"
server_key: "c9a5bc51e818e75a473d63fdd93d7cb9"
server_key_iv: "bff80b59dbe3cfdb89898db68406e435"
server_key_version: "encryptionkey@0"
I20240506 15:41:18.668072 29002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:41:18.685663 29002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:41:18.687918 29002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:41:18.688055 29002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:41:18.688302 29002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:41:18.688614 29002 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:41:18.688652 29002 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.688688 29002 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:41:18.688724 29002 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:41:18.694269 29002 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:35503
I20240506 15:41:18.694329 29130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:35503 every 8 connection(s)
I20240506 15:41:18.694633 29002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[28627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010078, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:41:18.702553 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 29002
I20240506 15:41:18.702661 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:41:18.702833 18371 external_mini_cluster.cc:1428] Setting key e38f967bc232cd706d1749d7f3175693
I20240506 15:41:18.703850 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.695482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:36627 (local address 127.17.240.254:38557)
0506 15:41:18.695580 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.695583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.695701 (+ 118us) server_negotiation.cc:407] Connection header received
0506 15:41:18.696210 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.696211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.696224 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.696280 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:18.697389 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.697879 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.698422 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.698600 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.700970 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:18.700976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:18.700976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:41:18.700984 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:18.702257 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:18.702710 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:18.702712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:18.702712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:18.702738 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:18.703064 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:18.703066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:18.703066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:18.703212 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:18.703290 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.703588 (+ 298us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.703721 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20240506 15:41:18.703933 29133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.695392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38557 (local address 127.17.240.195:36627)
0506 15:41:18.695578 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:18.695614 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:18.696100 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:18.696340 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:18.696344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:18.696696 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:41:18.697281 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.697286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.697967 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.697968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:18.698343 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.698344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.698479 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.699283 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:41:18.699301 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:41:18.700861 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:41:18.702338 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:41:18.702343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:41:18.702347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:41:18.702558 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:41:18.702798 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:41:18.702799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:41:18.702819 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:41:18.702918 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:41:18.703285 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:41:18.703290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:41:18.703434 (+ 144us) client_negotiation.cc:769] Sending connection context
0506 15:41:18.703555 (+ 121us) client_negotiation.cc:240] Negotiation successful
0506 15:41:18.703755 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20240506 15:41:18.704412 29131 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38557
I20240506 15:41:18.704504 29131 heartbeater.cc:464] Registering TS with master...
I20240506 15:41:18.704715 29131 heartbeater.cc:511] Master 127.17.240.254:38557 requested a full tablet report, sending...
I20240506 15:41:18.705205 28672 ts_manager.cc:194] Registered new tserver with Master: 99a62b226d1141cd89d1c925befcc9ca (127.17.240.195:35503)
I20240506 15:41:18.705621 28672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:36627
I20240506 15:41:18.714968 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:41:18.721994 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.716330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.17.240.254:38557)
0506 15:41:18.716407 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.716409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.716505 (+ 96us) server_negotiation.cc:407] Connection header received
0506 15:41:18.716565 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.716566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.716574 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.716628 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:41:18.717386 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.717887 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.718435 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.718533 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.719661 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:41:18.719664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:41:18.719664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:41:18.719670 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:41:18.721152 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:18.721456 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:18.721457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:18.721457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:18.721474 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:41:18.721584 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:41:18.721584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:41:18.721585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:41:18.721694 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:41:18.721769 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.721848 (+ 79us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.721912 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20240506 15:41:18.723589 28672 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:41:18.725030 28672 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:41:18.730549 29140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.727119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:35503 (local address 127.0.0.1:58814)
0506 15:41:18.727439 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:18.727466 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:18.727536 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:18.727766 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:18.727767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:18.727768 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:41:18.728066 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.728068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.729428 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.729429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:18.730267 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.730270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.730337 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.730374 (+ 37us) client_negotiation.cc:769] Sending connection context
0506 15:41:18.730424 (+ 50us) client_negotiation.cc:240] Negotiation successful
0506 15:41:18.730470 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":4,"thread_start_us":54,"threads_started":1}
I20240506 15:41:18.730909 29141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.727181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.17.240.193:42175)
0506 15:41:18.727346 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.727354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.727540 (+ 186us) server_negotiation.cc:407] Connection header received
0506 15:41:18.727924 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.727926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.727941 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.728026 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:41:18.728396 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.729102 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.730108 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.730621 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.730671 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.730788 (+ 117us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.730833 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20240506 15:41:18.731045 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.727298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.17.240.195:35503)
0506 15:41:18.727617 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.727624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.727639 (+ 15us) server_negotiation.cc:407] Connection header received
0506 15:41:18.727673 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.727675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.727690 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.727763 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:41:18.728163 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.729354 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.730350 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.730795 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.730857 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.730888 (+ 31us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.730941 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":128,"threads_started":1}
I20240506 15:41:18.731235 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.727119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:42175 (local address 127.0.0.1:54800)
0506 15:41:18.727437 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:18.727466 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:18.727832 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:18.728110 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:18.728111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:18.728114 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:41:18.728318 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.728320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.729176 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.729177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:18.729987 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.729989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.730698 (+ 709us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.730702 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:41:18.731131 (+ 429us) client_negotiation.cc:240] Negotiation successful
0506 15:41:18.731172 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":70,"threads_started":1}
I20240506 15:41:18.731659 29065 tablet_service.cc:1467] Processing CreateTablet for tablet 7a80891961824259a45fc7f202864e32 (DEFAULT_TABLE table=test-table [id=86b983789de14b1e9ff998bef4ab566e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:41:18.731873 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.727335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:43731 (local address 127.0.0.1:42244)
0506 15:41:18.727488 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:18.727497 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:18.727537 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:18.727756 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:18.727760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:18.727764 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:41:18.728066 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.728068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.730654 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.730654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:18.731663 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.731666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.731729 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.731733 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:41:18.731764 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:41:18.731804 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":52,"threads_started":1}
I20240506 15:41:18.732012 29065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a80891961824259a45fc7f202864e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:41:18.732664 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.727413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.17.240.194:43731)
0506 15:41:18.727571 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.727579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.727588 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:18.727629 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.727631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.727649 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.727739 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:41:18.728163 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.729345 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.731976 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.732506 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.732534 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.732561 (+ 27us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.732588 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20240506 15:41:18.732919 28797 tablet_service.cc:1467] Processing CreateTablet for tablet 7a80891961824259a45fc7f202864e32 (DEFAULT_TABLE table=test-table [id=86b983789de14b1e9ff998bef4ab566e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:41:18.733124 28797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a80891961824259a45fc7f202864e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:41:18.734545 29145 tablet_bootstrap.cc:492] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: Bootstrap starting.
I20240506 15:41:18.735121 29145 tablet_bootstrap.cc:654] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: Neither blocks nor log segments found. Creating new log.
I20240506 15:41:18.735275 29146 tablet_bootstrap.cc:492] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f: Bootstrap starting.
I20240506 15:41:18.735620 29145 log.cc:824] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: Log is configured to *not* fsync() on all Append() calls
I20240506 15:41:18.735569 28931 tablet_service.cc:1467] Processing CreateTablet for tablet 7a80891961824259a45fc7f202864e32 (DEFAULT_TABLE table=test-table [id=86b983789de14b1e9ff998bef4ab566e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:41:18.735836 28931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a80891961824259a45fc7f202864e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:41:18.735880 29146 tablet_bootstrap.cc:654] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f: Neither blocks nor log segments found. Creating new log.
I20240506 15:41:18.736163 29146 log.cc:824] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f: Log is configured to *not* fsync() on all Append() calls
I20240506 15:41:18.736395 29145 tablet_bootstrap.cc:492] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: No bootstrap required, opened a new log
I20240506 15:41:18.736452 29145 ts_tablet_manager.cc:1367] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:41:18.736702 29146 tablet_bootstrap.cc:492] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f: No bootstrap required, opened a new log
I20240506 15:41:18.736742 29146 ts_tablet_manager.cc:1367] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:41:18.738199 29146 raft_consensus.cc:348] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42175 } } peers { permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 } } peers { permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 } }
I20240506 15:41:18.738294 29146 raft_consensus.cc:374] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:41:18.738309 29146 raft_consensus.cc:729] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68e9e76d5ec24d1c9c4466d7446f5b7f, State: Initialized, Role: FOLLOWER
I20240506 15:41:18.738390 29146 consensus_queue.cc:260] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42175 } } peers { permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 } } peers { permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 } }
I20240506 15:41:18.738608 29146 ts_tablet_manager.cc:1398] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:41:18.738646 28863 heartbeater.cc:503] Master 127.17.240.254:38557 was elected leader, sending a full tablet report...
I20240506 15:41:18.738919 29150 tablet_bootstrap.cc:492] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: Bootstrap starting.
I20240506 15:41:18.738682 29145 raft_consensus.cc:348] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42175 } } peers { permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 } } peers { permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 } }
I20240506 15:41:18.739164 29145 raft_consensus.cc:374] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:41:18.739210 29145 raft_consensus.cc:729] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a62b226d1141cd89d1c925befcc9ca, State: Initialized, Role: FOLLOWER
I20240506 15:41:18.739307 29145 consensus_queue.cc:260] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42175 } } peers { permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 } } peers { permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 } }
I20240506 15:41:18.739526 29150 tablet_bootstrap.cc:654] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: Neither blocks nor log segments found. Creating new log.
I20240506 15:41:18.739564 29145 ts_tablet_manager.cc:1398] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240506 15:41:18.739718 29131 heartbeater.cc:503] Master 127.17.240.254:38557 was elected leader, sending a full tablet report...
I20240506 15:41:18.739822 29150 log.cc:824] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: Log is configured to *not* fsync() on all Append() calls
I20240506 15:41:18.740470 29150 tablet_bootstrap.cc:492] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: No bootstrap required, opened a new log
I20240506 15:41:18.740523 29150 ts_tablet_manager.cc:1367] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:41:18.742023 29150 raft_consensus.cc:348] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42175 } } peers { permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 } } peers { permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 } }
I20240506 15:41:18.742125 29150 raft_consensus.cc:374] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:41:18.742141 29150 raft_consensus.cc:729] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01691e975f84377ab86a51900758dc4, State: Initialized, Role: FOLLOWER
I20240506 15:41:18.742214 29150 consensus_queue.cc:260] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42175 } } peers { permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 } } peers { permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 } }
I20240506 15:41:18.742422 29150 ts_tablet_manager.cc:1398] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:41:18.742486 28997 heartbeater.cc:503] Master 127.17.240.254:38557 was elected leader, sending a full tablet report...
W20240506 15:41:18.783505 28998 tablet.cc:2369] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:41:18.833699 29149 raft_consensus.cc:482] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:41:18.833840 29149 raft_consensus.cc:504] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42175 } } peers { permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 } } peers { permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 } }
I20240506 15:41:18.834276 29149 leader_election.cc:288] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a62b226d1141cd89d1c925befcc9ca (127.17.240.195:35503), f01691e975f84377ab86a51900758dc4 (127.17.240.194:43731)
I20240506 15:41:18.836704 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.834494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:43731 (local address 127.17.240.193:37365)
0506 15:41:18.834605 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:18.834620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:18.834675 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:18.834867 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:18.834868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:18.834870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:41:18.835134 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.835137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.835838 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.835839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:18.836446 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.836448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.836519 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.836526 (+ 7us) client_negotiation.cc:769] Sending connection context
0506 15:41:18.836560 (+ 34us) client_negotiation.cc:240] Negotiation successful
0506 15:41:18.836608 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20240506 15:41:18.836714 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.834532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:35503 (local address 127.17.240.193:50409)
0506 15:41:18.834746 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:18.834757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:41:18.834819 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:41:18.834993 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:41:18.834994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:41:18.834995 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:41:18.835248 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.835250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.835906 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.835906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:41:18.836473 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:41:18.836474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.836551 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.836555 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:41:18.836587 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:41:18.836622 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":124,"threads_started":1}
I20240506 15:41:18.837162 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.834628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:37365 (local address 127.17.240.194:43731)
0506 15:41:18.834671 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.834673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.834683 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:41:18.834739 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.834740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.834750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.834821 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:41:18.835209 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.835753 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.836543 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.837063 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.837070 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.837090 (+ 20us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.837113 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:41:18.837162 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.834601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:50409 (local address 127.17.240.195:35503)
0506 15:41:18.834659 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.834662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.834824 (+ 162us) server_negotiation.cc:407] Connection header received
0506 15:41:18.834904 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.834906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.834917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.834983 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:41:18.835331 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.835797 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.836571 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.837039 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.837049 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.837072 (+ 23us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.837107 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:41:18.837466 28951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a80891961824259a45fc7f202864e32" candidate_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01691e975f84377ab86a51900758dc4" is_pre_election: true
I20240506 15:41:18.837474 29085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a80891961824259a45fc7f202864e32" candidate_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a62b226d1141cd89d1c925befcc9ca" is_pre_election: true
I20240506 15:41:18.837599 28951 raft_consensus.cc:2454] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68e9e76d5ec24d1c9c4466d7446f5b7f in term 0.
I20240506 15:41:18.837599 29085 raft_consensus.cc:2454] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68e9e76d5ec24d1c9c4466d7446f5b7f in term 0.
I20240506 15:41:18.837853 28752 leader_election.cc:302] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68e9e76d5ec24d1c9c4466d7446f5b7f, 99a62b226d1141cd89d1c925befcc9ca; no voters:
I20240506 15:41:18.838001 29149 raft_consensus.cc:2781] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:41:18.838061 29149 raft_consensus.cc:482] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:41:18.838078 29149 raft_consensus.cc:3037] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:41:18.838727 29149 raft_consensus.cc:504] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42175 } } peers { permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 } } peers { permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 } }
I20240506 15:41:18.838858 29149 leader_election.cc:288] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [CANDIDATE]: Term 1 election: Requested vote from peers 99a62b226d1141cd89d1c925befcc9ca (127.17.240.195:35503), f01691e975f84377ab86a51900758dc4 (127.17.240.194:43731)
I20240506 15:41:18.839057 29085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a80891961824259a45fc7f202864e32" candidate_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a62b226d1141cd89d1c925befcc9ca"
I20240506 15:41:18.839076 28951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a80891961824259a45fc7f202864e32" candidate_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01691e975f84377ab86a51900758dc4"
I20240506 15:41:18.839135 28951 raft_consensus.cc:3037] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:41:18.839140 29085 raft_consensus.cc:3037] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:41:18.839804 28951 raft_consensus.cc:2454] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68e9e76d5ec24d1c9c4466d7446f5b7f in term 1.
I20240506 15:41:18.839931 29085 raft_consensus.cc:2454] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68e9e76d5ec24d1c9c4466d7446f5b7f in term 1.
I20240506 15:41:18.840014 28750 leader_election.cc:302] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68e9e76d5ec24d1c9c4466d7446f5b7f, f01691e975f84377ab86a51900758dc4; no voters:
I20240506 15:41:18.840154 29149 raft_consensus.cc:2781] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:41:18.840319 29149 raft_consensus.cc:686] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [term 1 LEADER]: Becoming Leader. State: Replica: 68e9e76d5ec24d1c9c4466d7446f5b7f, State: Running, Role: LEADER
I20240506 15:41:18.840420 29149 consensus_queue.cc:237] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42175 } } peers { permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 } } peers { permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 } }
I20240506 15:41:18.841351 28672 catalog_manager.cc:5484] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 68e9e76d5ec24d1c9c4466d7446f5b7f (127.17.240.193). New cstate: current_term: 1 leader_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e9e76d5ec24d1c9c4466d7446f5b7f" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 } health_report { overall_health: UNKNOWN } } }
I20240506 15:41:18.846999 29141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.844831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.17.240.193:42175)
0506 15:41:18.844920 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0506 15:41:18.844922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:41:18.844931 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:41:18.844964 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:41:18.844964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:41:18.844980 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:41:18.845064 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:41:18.845325 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.845831 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:41:18.846467 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:41:18.846603 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:41:18.846638 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:41:18.846719 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:41:18.846778 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0506 15:41:18.846917 (+ 139us) server_negotiation.cc:299] Negotiation successful
0506 15:41:18.846941 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20240506 15:41:18.848837 28951 raft_consensus.cc:1261] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4 [term 1 FOLLOWER]: Refusing update from remote peer 68e9e76d5ec24d1c9c4466d7446f5b7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:41:18.848834 29085 raft_consensus.cc:1261] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca [term 1 FOLLOWER]: Refusing update from remote peer 68e9e76d5ec24d1c9c4466d7446f5b7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:41:18.849227 29155 consensus_queue.cc:1035] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a62b226d1141cd89d1c925befcc9ca" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:41:18.849371 29149 consensus_queue.cc:1035] T 7a80891961824259a45fc7f202864e32 P 68e9e76d5ec24d1c9c4466d7446f5b7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f01691e975f84377ab86a51900758dc4" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:41:18.851883 29160 mvcc.cc:205] Tried to move back new op lower bound from 7024681282962026496 to 7024681282930978816. Current Snapshot: MvccSnapshot[applied={T|T < 7024681282962026496}]
I20240506 15:41:18.852109 29159 mvcc.cc:205] Tried to move back new op lower bound from 7024681282962026496 to 7024681282930978816. Current Snapshot: MvccSnapshot[applied={T|T < 7024681282962026496}]
I20240506 15:41:18.853986 28671 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47446:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:41:18.854074 28671 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47446:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:41:18.854810 28671 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4273d8a5c8db41368decc5043fab74d7: Sending DeleteTablet for 3 replicas of tablet 7a80891961824259a45fc7f202864e32
I20240506 15:41:18.855166 29065 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a80891961824259a45fc7f202864e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:41:18 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:58814
I20240506 15:41:18.855228 28797 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a80891961824259a45fc7f202864e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:41:18 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:54800
I20240506 15:41:18.855407 28931 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a80891961824259a45fc7f202864e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:41:18 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:42244
I20240506 15:41:18.855650 29170 tablet_replica.cc:330] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: stopping tablet replica
I20240506 15:41:18.855746 29170 raft_consensus.cc:2229] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:41:18.855806 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 28734
I20240506 15:41:18.856451 29168 tablet_replica.cc:330] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: stopping tablet replica
I20240506 15:41:18.856539 29168 raft_consensus.cc:2229] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:41:18.856637 29168 raft_consensus.cc:2258] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:41:18.855845 29170 raft_consensus.cc:2258] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:41:18.857540 29168 ts_tablet_manager.cc:1875] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:41:18.857697 29170 ts_tablet_manager.cc:1875] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:41:18.859184 29168 ts_tablet_manager.cc:1888] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:41:18.859257 29168 log.cc:1196] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/7a80891961824259a45fc7f202864e32
I20240506 15:41:18.859524 29168 ts_tablet_manager.cc:1909] T 7a80891961824259a45fc7f202864e32 P 99a62b226d1141cd89d1c925befcc9ca: Deleting consensus metadata
I20240506 15:41:18.859701 29170 ts_tablet_manager.cc:1888] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:41:18.859778 29170 log.cc:1196] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/7a80891961824259a45fc7f202864e32
I20240506 15:41:18.859932 28658 catalog_manager.cc:4830] TS 99a62b226d1141cd89d1c925befcc9ca (127.17.240.195:35503): tablet 7a80891961824259a45fc7f202864e32 (table test-table [id=86b983789de14b1e9ff998bef4ab566e]) successfully deleted
I20240506 15:41:18.860072 29170 ts_tablet_manager.cc:1909] T 7a80891961824259a45fc7f202864e32 P f01691e975f84377ab86a51900758dc4: Deleting consensus metadata
I20240506 15:41:18.860633 28656 catalog_manager.cc:4830] TS f01691e975f84377ab86a51900758dc4 (127.17.240.194:43731): tablet 7a80891961824259a45fc7f202864e32 (table test-table [id=86b983789de14b1e9ff998bef4ab566e]) successfully deleted
W20240506 15:41:18.862711 28656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:42175 (error 108)
I20240506 15:41:18.862954 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 28868
I20240506 15:41:18.863620 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:41:18.863112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:42175 (local address 127.0.0.1:54808)
0506 15:41:18.863543 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0506 15:41:18.863580 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:42175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411}
W20240506 15:41:18.867193 28656 catalog_manager.cc:4573] TS 68e9e76d5ec24d1c9c4466d7446f5b7f (127.17.240.193:42175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a80891961824259a45fc7f202864e32: Network error: Client connection negotiation failed: client connection to 127.17.240.193:42175: connect: Connection refused (error 111)
I20240506 15:41:18.868445 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 29002
I20240506 15:41:18.873453 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 28640
2024-05-06T15:41:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1890 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[29174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[29174](info): set up 2 sockets
May 06 15:41:18 dist-test-slave-v0k1 krb5kdc[29174](info): commencing operation
krb5kdc: starting...
W20240506 15:41:19.831328 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:41:19 dist-test-slave-v0k1 krb5kdc[29174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:41:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:41:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/rangeradmin_127.17.240.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/rangeradmin_127.17.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/rangerlookup_127.17.240.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/rangerlookup_127.17.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/HTTP_127.17.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/HTTP_127.17.240.212@KRBTEST.COM.keytab.
I20240506 15:41:20.003772 18371 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskb5AjvH/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/postgres -l logfile start
2024-05-06 15:41:22.823 UTC [29204] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-06 15:41:22.823 UTC [29204] LOG: listening on IPv4 address "127.17.240.212", port 41443
2024-05-06 15:41:22.827 UTC [29204] LOG: listening on Unix socket "/tmp/.s.PGSQL.41443"
2024-05-06 15:41:22.849 UTC [29207] LOG: database system was shut down at 2024-05-06 15:41:21 UTC
2024-05-06 15:41:22.854 UTC [29204] LOG: database system is ready to accept connections
I20240506 15:41:23.723146 18371 mini_postgres.cc:96] Postgres bound to 41443
2024-05-06 15:41:23.728 UTC [29215] FATAL: database "slave" does not exist
127.17.240.212:41443 - accepting connections
I20240506 15:41:23.729312 18371 mini_ranger.cc:161] Starting Ranger...
I20240506 15:41:23.743865 18371 mini_ranger.cc:84] Created miniranger Postgres user
I20240506 15:41:24.339413 18371 mini_ranger.cc:87] Created ranger Postgres database
I20240506 15:41:24.339491 18371 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-admin
2024-05-06 15:41:24,566 [I] DB FLAVOR :POSTGRES
2024-05-06 15:41:24,567 [I] --------- Verifying Ranger DB connection ---------
2024-05-06 15:41:24,567 [I] Checking connection..
2024-05-06 15:41:24,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-06 15:41:24,825 [I] Checking connection passed.
2024-05-06 15:41:24,825 [I] --------- Verifying version history table ---------
2024-05-06 15:41:24,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-06 15:41:25,081 [I] Table x_db_version_h does not exist in database ranger
2024-05-06 15:41:25,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-06 15:41:25,316 [I] Table x_db_version_h does not exist in database ranger
2024-05-06 15:41:25,316 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-06 15:41:25,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-06 15:41:25.569 UTC [29279] WARNING: there is no transaction in progress
2024-05-06 15:41:25,576 [I] create_dbversion_catalog.sql file imported successfully
2024-05-06 15:41:25,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-06 15:41:25,838 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-06 15:41:25,839 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-06 15:41:25,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-06 15:41:26,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-06 15:41:26,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v0k1.c.gcp-upstream.internal','N') ;"
2024-05-06 15:41:26,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-06 15:41:26,837 [I] Table x_portal_user does not exist in database ranger
2024-05-06 15:41:26,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-06 15:41:27,092 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-06 15:41:27,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-06 15:41:27,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-06 15:41:27,579 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-06 15:41:27,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-06 15:41:28.635 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.655 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.676 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.698 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.717 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.739 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.802 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.810 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.827 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.838 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.850 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.865 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.878 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.887 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.896 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:28.904 UTC [29387] WARNING: there is no transaction in progress
2024-05-06 15:41:29,668 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-06 15:41:29,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-06 15:41:29,921 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-06 15:41:29,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-06 15:41:30,181 [I] Table x_portal_user already exists in database 'ranger'
2024-05-06 15:41:30,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-06 15:41:30,446 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-06 15:41:30,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-06 15:41:30,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-06 15:41:30,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v0k1.c.gcp-upstream.internal';"
2024-05-06 15:41:31,184 [I] CORE_DB_SCHEMA import status has been updated
2024-05-06 15:41:31,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-06 15:41:31,419 [I] DB_PATCHES have already been applied
I20240506 15:41:31.425468 18371 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-admin:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskb5AjvH/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240506 15:41:31.425542 18371 mini_ranger.cc:193] Using host: 127.17.240.212
I20240506 15:41:31.428297 18371 mini_ranger.cc:227] Ranger admin URL: http://127.17.240.212:55721
May 06, 2024 3:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 06, 2024 3:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 06, 2024 3:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-admin/ews/webapp
May 06, 2024 3:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 06, 2024 3:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-admin/ews/webapp] .....
May 06, 2024 3:41:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-admin/ews/webapp].
May 06, 2024 3:41:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240506 15:41:32.464171 18371 mini_ranger.cc:160] Time spent starting Ranger: real 8.735s user 0.000s sys 0.009s
May 06, 2024 3:41:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 06, 2024 3:41:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
May 06, 2024 3:41:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 06, 2024 3:41:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 06, 2024 3:41:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240506 15:41:53.419198 18371 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/rangerkms_127.17.240.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/rangerkms_127.17.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.240.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.240.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.240.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/HTTP_127.17.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.240.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/HTTP_127.17.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 06 15:41:53 dist-test-slave-v0k1 krb5kdc[29174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010113, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240506 15:41:53.503154 18371 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240506 15:41:53.512334 18371 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240506 15:41:54.630196 18371 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240506 15:41:54.630280 18371 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-kms
I20240506 15:41:54.630302 18371 mini_ranger_kms.cc:227] Using postgres at 127.17.240.212:41443
2024-05-06 15:41:54,803 [I] DB FLAVOR :POSTGRES
2024-05-06 15:41:54,803 [I] --------- Verifying Ranger DB connection ---------
2024-05-06 15:41:54,803 [I] Checking connection
2024-05-06 15:41:54,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-06 15:41:55,074 [I] connection success
2024-05-06 15:41:55,074 [I] --------- Verifying Ranger DB tables ---------
2024-05-06 15:41:55,074 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-06 15:41:55,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-06 15:41:55,321 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-06 15:41:55,321 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-06 15:41:55,322 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-06 15:41:55,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:41443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-06 15:41:55,630 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240506 15:41:56.158572 18371 mini_ranger_kms.cc:310] Created kms service
I20240506 15:41:56.357576 18371 mini_ranger_kms.cc:326] Created kudu user
I20240506 15:41:56.404875 18371 mini_ranger_kms.cc:343] Created rangerkms user
May 06 15:41:56 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010113, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:41:56.854635 18371 mini_ranger_kms.cc:384] Added ranger policy
I20240506 15:41:56.854763 18371 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-kms:/tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240506 15:41:56.854792 18371 mini_ranger_kms.cc:240] Using host: 127.17.240.212
I20240506 15:41:56.857631 18371 mini_ranger_kms.cc:276] Ranger KMS PID: 29567
I20240506 15:41:56.857730 18371 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.240.212:50529
May 06, 2024 3:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 06, 2024 3:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 06, 2024 3:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 06, 2024 3:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-kms/ews/webapp] .....
May 06, 2024 3:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-kms/ews/webapp].
May 06, 2024 3:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240506 15:41:59.076473 18371 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.573s user 0.006s sys 0.149s
I20240506 15:41:59.076583 18371 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240506 15:41:59.076637 18371 mini_ranger_kms.cc:401] 127.17.240.212:50529/kms/v1/keys
May 06 15:41:59 dist-test-slave-v0k1 krb5kdc[29174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1715010119, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.240.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 06 15:42:00 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010119, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.240.212@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
May 06 15:42:00 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010119, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.240.212@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
May 06 15:42:03 dist-test-slave-v0k1 krb5kdc[29174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:42:03.041707 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39925
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:39925
--ranger_config_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:50529/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:03.122134 29618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:03.122344 29618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:03.122365 29618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:03.123750 29618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:42:03.123804 29618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:03.123818 29618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:03.123831 29618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:42:03.123842 29618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:42:03.125602 29618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.240.212:50529/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:39925
--ranger_config_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:03.125830 29618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:03.126024 29618 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:03.130581 29624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:03.131101 29623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:03.131826 29626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:03.132597 29618 server_base.cc:1034] running on GCE node
I20240506 15:42:03.132831 29618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:03.133082 29618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:03.134270 29618 hybrid_clock.cc:648] HybridClock initialized: now 1715010123134249 us; error 39 us; skew 500 ppm
May 06 15:42:03 dist-test-slave-v0k1 krb5kdc[29174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010123, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:03.137176 29618 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:42:03.137583 29618 webserver.cc:460] Webserver started at http://127.17.240.254:42233/ using document root <none> and password file <none>
I20240506 15:42:03.137758 29618 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:03.137797 29618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:03.137909 29618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 06 15:42:03 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010123, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:03.190694 29618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "8b172981d9b647058c5e8eaf5cf2ec18"
format_stamp: "Formatted at 2024-05-06 15:42:03 on dist-test-slave-v0k1"
server_key: "7229ed48b8684a53ea51f35a9e8ab5a3"
server_key_iv: "94306b32423859967a4471c96071f46a"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:03.191164 29618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "8b172981d9b647058c5e8eaf5cf2ec18"
format_stamp: "Formatted at 2024-05-06 15:42:03 on dist-test-slave-v0k1"
server_key: "7229ed48b8684a53ea51f35a9e8ab5a3"
server_key_iv: "94306b32423859967a4471c96071f46a"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:03.192829 29618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:42:03.208130 29633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:03.208459 29618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240506 15:42:03.208564 29618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "8b172981d9b647058c5e8eaf5cf2ec18"
format_stamp: "Formatted at 2024-05-06 15:42:03 on dist-test-slave-v0k1"
server_key: "7229ed48b8684a53ea51f35a9e8ab5a3"
server_key_iv: "94306b32423859967a4471c96071f46a"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:03.208667 29618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:03.222057 29618 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:03.224176 29618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:03.224282 29618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:03.228296 29618 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:39925
I20240506 15:42:03.228343 29685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:39925 every 8 connection(s)
I20240506 15:42:03.228637 29618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:42:03.229211 29686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:03.231426 29686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18: Bootstrap starting.
I20240506 15:42:03.231976 29686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:03.232226 29686 log.cc:824] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:03.232894 29686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18: No bootstrap required, opened a new log
I20240506 15:42:03.234671 29686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b172981d9b647058c5e8eaf5cf2ec18" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39925 } }
I20240506 15:42:03.234794 29686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:03.234812 29686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b172981d9b647058c5e8eaf5cf2ec18, State: Initialized, Role: FOLLOWER
I20240506 15:42:03.234920 29686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b172981d9b647058c5e8eaf5cf2ec18" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39925 } }
I20240506 15:42:03.234988 29686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:42:03.235049 29686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:42:03.235085 29686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:03.235611 29686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b172981d9b647058c5e8eaf5cf2ec18" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39925 } }
I20240506 15:42:03.235704 29686 leader_election.cc:302] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b172981d9b647058c5e8eaf5cf2ec18; no voters:
I20240506 15:42:03.235855 29686 leader_election.cc:288] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:42:03.235905 29689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:42:03.236070 29689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [term 1 LEADER]: Becoming Leader. State: Replica: 8b172981d9b647058c5e8eaf5cf2ec18, State: Running, Role: LEADER
I20240506 15:42:03.236076 29686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:42:03.236198 29689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b172981d9b647058c5e8eaf5cf2ec18" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39925 } }
I20240506 15:42:03.236524 29691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b172981d9b647058c5e8eaf5cf2ec18. Latest consensus state: current_term: 1 leader_uuid: "8b172981d9b647058c5e8eaf5cf2ec18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b172981d9b647058c5e8eaf5cf2ec18" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39925 } } }
I20240506 15:42:03.236598 29691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:03.236717 29690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b172981d9b647058c5e8eaf5cf2ec18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b172981d9b647058c5e8eaf5cf2ec18" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39925 } } }
I20240506 15:42:03.236752 29690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18 [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:03.236864 29694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:42:03.237300 29694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:42:03.237332 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 29618
I20240506 15:42:03.237439 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:42:03.237624 18371 external_mini_cluster.cc:1428] Setting key 5803c76292426079c07bd970b4a09f89
I20240506 15:42:03.238333 29686 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240506 15:42:03.238759 29694 catalog_manager.cc:1347] Generated new cluster ID: e0bc013352b6422d83e7be578b069bf5
I20240506 15:42:03.240317 29694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:42:03 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:03.254038 29694 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:42:03.254923 29694 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:42:03.259146 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:03.238294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.17.240.254:39925)
0506 15:42:03.238733 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:03.238746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:03.240148 (+ 1402us) server_negotiation.cc:407] Connection header received
0506 15:42:03.240595 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:03.240604 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:03.240825 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:03.241004 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:03.241795 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:03.242632 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:03.243816 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:03.244093 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:03.248930 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:03.248938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:03.248938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:03.248969 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:03.253891 (+ 4922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:03.257042 (+ 3151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:03.257045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:03.257046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:03.257105 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:03.257922 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:03.257923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:03.257923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:03.258189 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:03.258331 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:03.258699 (+ 368us) server_negotiation.cc:299] Negotiation successful
0506 15:42:03.258850 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":65,"threads_started":1}
I20240506 15:42:03.259876 29694 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18: Generated new TSK 0
I20240506 15:42:03.260071 29694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:42:03.983357 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:50529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39925
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:04.090471 29731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:04.090665 29731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:04.090689 29731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:04.092079 29731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:04.092128 29731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:04.092155 29731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:42:04.093876 29731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.240.212:50529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:04.094110 29731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:04.094321 29731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:04.099625 29739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:04.100322 29736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:04.100936 29737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:04.101526 29731 server_base.cc:1034] running on GCE node
I20240506 15:42:04.101707 29731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:04.101939 29731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:04.103070 29731 hybrid_clock.cc:648] HybridClock initialized: now 1715010124103040 us; error 43 us; skew 500 ppm
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:04.106261 29731 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:42:04.106719 29731 webserver.cc:460] Webserver started at http://127.17.240.193:40059/ using document root <none> and password file <none>
I20240506 15:42:04.106937 29731 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:04.106992 29731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:04.107172 29731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:04.124872 29731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "7ed9dec23763460fac696147f0dfd664"
format_stamp: "Formatted at 2024-05-06 15:42:04 on dist-test-slave-v0k1"
server_key: "081ce4dd505cd7ab2f180de6be758448"
server_key_iv: "14b4bc9cd9518b43b81aae313fb27cf7"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:04.125365 29731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "7ed9dec23763460fac696147f0dfd664"
format_stamp: "Formatted at 2024-05-06 15:42:04 on dist-test-slave-v0k1"
server_key: "081ce4dd505cd7ab2f180de6be758448"
server_key_iv: "14b4bc9cd9518b43b81aae313fb27cf7"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:04.127156 29731 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20240506 15:42:04.142120 29748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:04.142390 29731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240506 15:42:04.142496 29731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "7ed9dec23763460fac696147f0dfd664"
format_stamp: "Formatted at 2024-05-06 15:42:04 on dist-test-slave-v0k1"
server_key: "081ce4dd505cd7ab2f180de6be758448"
server_key_iv: "14b4bc9cd9518b43b81aae313fb27cf7"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:04.142590 29731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:04.176541 29731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:04.189597 29731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:04.189739 29731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:04.189939 29731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:42:04.190284 29731 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:42:04.190332 29731 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:04.190363 29731 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:42:04.190387 29731 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:04.197290 29731 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:44873
I20240506 15:42:04.197315 29862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:44873 every 8 connection(s)
I20240506 15:42:04.197657 29731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:42:04.200043 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 29731
I20240506 15:42:04.200153 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:42:04.200317 18371 external_mini_cluster.cc:1428] Setting key 2236cef77a76fd81053227cc945fae62
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:04.208467 29865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.198346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39925 (local address 127.17.240.193:48269)
0506 15:42:04.198828 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:04.198861 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:04.199924 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:04.200320 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:04.200325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:04.200656 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:04.201108 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.201115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.202158 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.202159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:04.202689 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.202691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.202824 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.203454 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:04.203476 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:04.205061 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:04.207114 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:04.207119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:04.207122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:04.207390 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:04.207556 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:04.207557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:04.207557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:04.207626 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:04.207865 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:04.207869 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:04.208026 (+ 157us) client_negotiation.cc:769] Sending connection context
0506 15:42:04.208137 (+ 111us) client_negotiation.cc:240] Negotiation successful
0506 15:42:04.208259 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":71,"threads_started":1}
I20240506 15:42:04.209210 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.198620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:48269 (local address 127.17.240.254:39925)
0506 15:42:04.198764 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:04.198766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:04.200089 (+ 1323us) server_negotiation.cc:407] Connection header received
0506 15:42:04.200121 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:04.200123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:04.200140 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:04.200196 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:04.201217 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.202058 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.203096 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.203277 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.205160 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:04.205165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:04.205165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:04.205172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:04.207029 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:04.207471 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:04.207472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:04.207473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:04.207498 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:04.207679 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:04.207679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:04.207680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:04.207782 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:04.207842 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:04.209009 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0506 15:42:04.209097 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20240506 15:42:04.209758 29863 heartbeater.cc:347] Connected to a master server at 127.17.240.254:39925
I20240506 15:42:04.209851 29863 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:04.210078 29863 heartbeater.cc:511] Master 127.17.240.254:39925 requested a full tablet report, sending...
I20240506 15:42:04.210624 29650 ts_manager.cc:194] Registered new tserver with Master: 7ed9dec23763460fac696147f0dfd664 (127.17.240.193:44873)
I20240506 15:42:04.211594 29650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:48269
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:42:04.243291 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:50529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39925
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20240506 15:42:04.350179 29870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:04.350381 29870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:04.350420 29870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:04.351871 29870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:04.351926 29870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:04.351972 29870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:42:04.353729 29870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.240.212:50529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:04.353960 29870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:04.354144 29870 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:04.359429 29875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:04.360101 29878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:04.360605 29876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:04.361394 29870 server_base.cc:1034] running on GCE node
I20240506 15:42:04.361598 29870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:04.361881 29870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:04.363073 29870 hybrid_clock.cc:648] HybridClock initialized: now 1715010124363050 us; error 47 us; skew 500 ppm
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:04.366266 29870 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:42:04.366771 29870 webserver.cc:460] Webserver started at http://127.17.240.194:36647/ using document root <none> and password file <none>
I20240506 15:42:04.366998 29870 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:04.367100 29870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:04.367254 29870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:04.386735 29870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "4d8402d498b248649e33f9cb69ac9ceb"
format_stamp: "Formatted at 2024-05-06 15:42:04 on dist-test-slave-v0k1"
server_key: "7477f2b9ef61d55adc97e695b54787a9"
server_key_iv: "0f2c2f77b00ec7ca9c7ad63e8238e3b3"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:04.387274 29870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "4d8402d498b248649e33f9cb69ac9ceb"
format_stamp: "Formatted at 2024-05-06 15:42:04 on dist-test-slave-v0k1"
server_key: "7477f2b9ef61d55adc97e695b54787a9"
server_key_iv: "0f2c2f77b00ec7ca9c7ad63e8238e3b3"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:04.389192 29870 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20240506 15:42:04.404458 29886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:04.404737 29870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240506 15:42:04.404843 29870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "4d8402d498b248649e33f9cb69ac9ceb"
format_stamp: "Formatted at 2024-05-06 15:42:04 on dist-test-slave-v0k1"
server_key: "7477f2b9ef61d55adc97e695b54787a9"
server_key_iv: "0f2c2f77b00ec7ca9c7ad63e8238e3b3"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:04.404945 29870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:04.416934 29870 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:04.419590 29870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:04.419775 29870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:04.419988 29870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:42:04.420316 29870 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:42:04.420363 29870 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:04.420394 29870 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:42:04.420413 29870 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:04.429035 29870 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:33809
I20240506 15:42:04.429476 29870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:42:04.430838 29999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:33809 every 8 connection(s)
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:04.439707 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 29870
I20240506 15:42:04.439826 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:42:04.440030 18371 external_mini_cluster.cc:1428] Setting key 5e5dd893c54bff70f6bdccbf9f6dad83
I20240506 15:42:04.441154 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.430737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:46437 (local address 127.17.240.254:39925)
0506 15:42:04.430795 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:04.430797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:04.431156 (+ 359us) server_negotiation.cc:407] Connection header received
0506 15:42:04.432123 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:04.432124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:04.432140 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:04.432197 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:04.432902 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.433691 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.434502 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.434691 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.437384 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:04.437388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:04.437389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:04.437396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:04.439150 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:04.439601 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:04.439602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:04.439602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:04.439629 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:04.439834 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:04.439835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:04.439835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:04.439957 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:04.439998 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:04.440941 (+ 943us) server_negotiation.cc:299] Negotiation successful
0506 15:42:04.441040 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240506 15:42:04.441586 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.430499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39925 (local address 127.17.240.194:46437)
0506 15:42:04.431052 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:04.431082 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:04.432010 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:04.432304 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:04.432308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:04.432506 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:04.432795 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.432799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.433801 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.433802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:04.434399 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.434402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.434814 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.435514 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:04.435535 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:04.437278 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:04.439239 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:04.439244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:04.439247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:04.439508 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:04.439693 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:04.439694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:04.439694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:04.439772 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:04.440042 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:04.440047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:04.440755 (+ 708us) client_negotiation.cc:769] Sending connection context
0506 15:42:04.441267 (+ 512us) client_negotiation.cc:240] Negotiation successful
0506 15:42:04.441393 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":65,"threads_started":1}
I20240506 15:42:04.441967 30000 heartbeater.cc:347] Connected to a master server at 127.17.240.254:39925
I20240506 15:42:04.442052 30000 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:04.442262 30000 heartbeater.cc:511] Master 127.17.240.254:39925 requested a full tablet report, sending...
I20240506 15:42:04.442593 29650 ts_manager.cc:194] Registered new tserver with Master: 4d8402d498b248649e33f9cb69ac9ceb (127.17.240.194:33809)
I20240506 15:42:04.443121 29650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:46437
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:42:04.491917 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:50529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39925
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:04.585848 30005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:04.586213 30005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:04.586308 30005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:04.588605 30005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:04.588688 30005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:04.588729 30005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:42:04.591388 30005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.240.212:50529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:04.591696 30005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:04.592003 30005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:04.597932 30011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:04.598362 30010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:04.599180 30013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:04.599874 30005 server_base.cc:1034] running on GCE node
I20240506 15:42:04.600118 30005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:04.600365 30005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:04.603582 30005 hybrid_clock.cc:648] HybridClock initialized: now 1715010124603553 us; error 48 us; skew 500 ppm
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:04.606982 30005 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:42:04.607558 30005 webserver.cc:460] Webserver started at http://127.17.240.195:37941/ using document root <none> and password file <none>
I20240506 15:42:04.607791 30005 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:04.607842 30005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:04.608007 30005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:04.633919 30005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "2136c4620b9f43dda3fb16e4c56e5597"
format_stamp: "Formatted at 2024-05-06 15:42:04 on dist-test-slave-v0k1"
server_key: "f26152a5aecd13839f276bf2a3d07550"
server_key_iv: "676374c59cff0f4a727f8a5ac31de281"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:04.634485 30005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "2136c4620b9f43dda3fb16e4c56e5597"
format_stamp: "Formatted at 2024-05-06 15:42:04 on dist-test-slave-v0k1"
server_key: "f26152a5aecd13839f276bf2a3d07550"
server_key_iv: "676374c59cff0f4a727f8a5ac31de281"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:04.643208 30005 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.002s
I20240506 15:42:04.657405 30020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:04.657675 30005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20240506 15:42:04.657756 30005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "2136c4620b9f43dda3fb16e4c56e5597"
format_stamp: "Formatted at 2024-05-06 15:42:04 on dist-test-slave-v0k1"
server_key: "f26152a5aecd13839f276bf2a3d07550"
server_key_iv: "676374c59cff0f4a727f8a5ac31de281"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:04.657835 30005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:04.684962 30005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:04.687569 30005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:04.687776 30005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:04.688047 30005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:42:04.688422 30005 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:42:04.688472 30005 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:04.688500 30005 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:42:04.688522 30005 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:04.696166 30005 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:40303
I20240506 15:42:04.696648 30005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:42:04.697741 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 30005
I20240506 15:42:04.697865 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:42:04.698063 18371 external_mini_cluster.cc:1428] Setting key d84b788f84e739a9b50d41d889fa5f7a
I20240506 15:42:04.698333 30133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:40303 every 8 connection(s)
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:04.709110 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.697915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:60283 (local address 127.17.240.254:39925)
0506 15:42:04.697963 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:04.697965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:04.698665 (+ 700us) server_negotiation.cc:407] Connection header received
0506 15:42:04.699362 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:04.699363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:04.699376 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:04.699404 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:04.700520 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.702404 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.703217 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.703381 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.705911 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:04.705916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:04.705916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:04.705924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:04.707675 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:04.708135 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:04.708136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:04.708137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:04.708160 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:04.708345 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:04.708346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:04.708346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:04.708451 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:04.708506 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:04.708898 (+ 392us) server_negotiation.cc:299] Negotiation successful
0506 15:42:04.708985 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:42:04.709818 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.697660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39925 (local address 127.17.240.195:60283)
0506 15:42:04.698551 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:04.698580 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:04.699243 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:04.699595 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:04.699599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:04.699872 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:04.700412 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.700418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.702526 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.702527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:04.703125 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.703128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.703514 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.704179 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:04.704203 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:04.705794 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:04.707768 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:04.707774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:04.707778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:04.708055 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:04.708219 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:04.708220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:04.708220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:04.708289 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:04.708573 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:04.708578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:04.708730 (+ 152us) client_negotiation.cc:769] Sending connection context
0506 15:42:04.709462 (+ 732us) client_negotiation.cc:240] Negotiation successful
0506 15:42:04.709618 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":63,"threads_started":1}
I20240506 15:42:04.710393 30134 heartbeater.cc:347] Connected to a master server at 127.17.240.254:39925
I20240506 15:42:04.710489 30134 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:04.710690 30134 heartbeater.cc:511] Master 127.17.240.254:39925 requested a full tablet report, sending...
I20240506 15:42:04.711180 29650 ts_manager.cc:194] Registered new tserver with Master: 2136c4620b9f43dda3fb16e4c56e5597 (127.17.240.195:40303)
I20240506 15:42:04.711674 29650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:60283
I20240506 15:42:04.722136 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:42:04.733597 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.726087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.17.240.254:39925)
0506 15:42:04.726144 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:04.726146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:04.726435 (+ 289us) server_negotiation.cc:407] Connection header received
0506 15:42:04.726475 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:04.726476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:04.726488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:04.726553 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:04.727500 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.728281 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.729079 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.729252 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.730705 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:04.730709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:04.730709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:04.730717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:04.732517 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:04.732872 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:04.732874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:04.732874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:04.732900 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:04.733051 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:04.733052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:04.733052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:04.733168 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:04.733277 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:04.733396 (+ 119us) server_negotiation.cc:299] Negotiation successful
0506 15:42:04.733492 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240506 15:42:04.735657 29650 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:42:04.737628 29650 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:42:04.750885 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.740195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:33809 (local address 127.0.0.1:33494)
0506 15:42:04.740482 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:04.740510 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:04.740576 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:04.741188 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:04.741191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:04.741194 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:42:04.741447 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.741449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.749315 (+ 7866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.749318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:04.750550 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.750552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.750624 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.750677 (+ 53us) client_negotiation.cc:769] Sending connection context
0506 15:42:04.750733 (+ 56us) client_negotiation.cc:240] Negotiation successful
0506 15:42:04.750785 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
I20240506 15:42:04.752010 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.740273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.17.240.194:33809)
0506 15:42:04.740929 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:04.740937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:04.740946 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:42:04.740992 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:04.740993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:04.741011 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:04.741089 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:42:04.741543 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.749213 (+ 7670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.751253 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.751794 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.751849 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:04.751894 (+ 45us) server_negotiation.cc:299] Negotiation successful
0506 15:42:04.751932 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":63,"threads_started":1}
I20240506 15:42:04.752871 29934 tablet_service.cc:1467] Processing CreateTablet for tablet 879aef5d4c1345b6a7a51b6f81d1bf4d (DEFAULT_TABLE table=test-table [id=8692a7027fa940a99f98ffc6f75e73c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:42:04.753190 29934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879aef5d4c1345b6a7a51b6f81d1bf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:04.753185 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.740606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:44873 (local address 127.0.0.1:56944)
0506 15:42:04.742950 (+ 2344us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:04.742958 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:04.743000 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:04.749660 (+ 6660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:04.749662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:04.749665 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:42:04.749894 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.749897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.752023 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.752024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:04.752963 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.752966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.753040 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.753044 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:42:04.753076 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:42:04.753113 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2332,"thread_start_us":61,"threads_started":1}
I20240506 15:42:04.754129 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.740695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.17.240.193:44873)
0506 15:42:04.749384 (+ 8689us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:04.749389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:04.749400 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:42:04.749446 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:04.749449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:04.749477 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:04.749565 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:42:04.749972 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.751049 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.753404 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.753925 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.753980 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:04.754012 (+ 32us) server_negotiation.cc:299] Negotiation successful
0506 15:42:04.754050 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8650,"thread_start_us":49,"threads_started":1}
I20240506 15:42:04.754742 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.740428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:40303 (local address 127.0.0.1:45514)
0506 15:42:04.742804 (+ 2376us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:04.742815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:04.742873 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:04.750561 (+ 7688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:04.750562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:04.750565 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:42:04.750774 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.750776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.753679 (+ 2903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.753680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:04.754558 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.754561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.754609 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.754612 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:42:04.754634 (+ 22us) client_negotiation.cc:240] Negotiation successful
0506 15:42:04.754670 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2358,"thread_start_us":87,"threads_started":1}
I20240506 15:42:04.755843 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.742571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.17.240.195:40303)
0506 15:42:04.750320 (+ 7749us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:04.750328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:04.750338 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:42:04.750376 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:04.750378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:04.750394 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:04.750483 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:42:04.750844 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.751932 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.754953 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.755509 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.755552 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:04.755587 (+ 35us) server_negotiation.cc:299] Negotiation successful
0506 15:42:04.755630 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7710,"thread_start_us":103,"threads_started":1}
I20240506 15:42:04.756388 30060 tablet_service.cc:1467] Processing CreateTablet for tablet 879aef5d4c1345b6a7a51b6f81d1bf4d (DEFAULT_TABLE table=test-table [id=8692a7027fa940a99f98ffc6f75e73c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:42:04.756700 30060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879aef5d4c1345b6a7a51b6f81d1bf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:04.757282 29794 tablet_service.cc:1467] Processing CreateTablet for tablet 879aef5d4c1345b6a7a51b6f81d1bf4d (DEFAULT_TABLE table=test-table [id=8692a7027fa940a99f98ffc6f75e73c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:42:04.757544 29794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879aef5d4c1345b6a7a51b6f81d1bf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:04.760352 30153 tablet_bootstrap.cc:492] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: Bootstrap starting.
I20240506 15:42:04.761008 30153 tablet_bootstrap.cc:654] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:04.761211 30154 tablet_bootstrap.cc:492] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664: Bootstrap starting.
I20240506 15:42:04.761627 30153 log.cc:824] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:04.761910 30154 tablet_bootstrap.cc:654] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:04.762162 30152 tablet_bootstrap.cc:492] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: Bootstrap starting.
I20240506 15:42:04.762406 30153 tablet_bootstrap.cc:492] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: No bootstrap required, opened a new log
I20240506 15:42:04.762488 30153 ts_tablet_manager.cc:1367] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:42:04.763077 30152 tablet_bootstrap.cc:654] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:04.763304 30154 log.cc:824] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:04.764065 30152 log.cc:824] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:04.764580 30153 raft_consensus.cc:348] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8402d498b248649e33f9cb69ac9ceb" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33809 } } peers { permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 } } peers { permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 } }
I20240506 15:42:04.764726 30153 raft_consensus.cc:374] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:04.764752 30153 raft_consensus.cc:729] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2136c4620b9f43dda3fb16e4c56e5597, State: Initialized, Role: FOLLOWER
I20240506 15:42:04.764844 30154 tablet_bootstrap.cc:492] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664: No bootstrap required, opened a new log
I20240506 15:42:04.764842 30153 consensus_queue.cc:260] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8402d498b248649e33f9cb69ac9ceb" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33809 } } peers { permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 } } peers { permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 } }
I20240506 15:42:04.764916 30154 ts_tablet_manager.cc:1367] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240506 15:42:04.765096 30153 ts_tablet_manager.cc:1398] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:42:04.765270 30134 heartbeater.cc:503] Master 127.17.240.254:39925 was elected leader, sending a full tablet report...
I20240506 15:42:04.767112 30152 tablet_bootstrap.cc:492] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: No bootstrap required, opened a new log
I20240506 15:42:04.767225 30152 ts_tablet_manager.cc:1367] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240506 15:42:04.767335 30154 raft_consensus.cc:348] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8402d498b248649e33f9cb69ac9ceb" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33809 } } peers { permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 } } peers { permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 } }
I20240506 15:42:04.767484 30154 raft_consensus.cc:374] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:04.767557 30154 raft_consensus.cc:729] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed9dec23763460fac696147f0dfd664, State: Initialized, Role: FOLLOWER
I20240506 15:42:04.767715 30154 consensus_queue.cc:260] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8402d498b248649e33f9cb69ac9ceb" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33809 } } peers { permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 } } peers { permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 } }
I20240506 15:42:04.768024 30154 ts_tablet_manager.cc:1398] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:42:04.768231 29863 heartbeater.cc:503] Master 127.17.240.254:39925 was elected leader, sending a full tablet report...
May 06 15:42:04 dist-test-slave-v0k1 krb5kdc[29174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010124, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:04.769997 30152 raft_consensus.cc:348] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8402d498b248649e33f9cb69ac9ceb" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33809 } } peers { permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 } } peers { permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 } }
I20240506 15:42:04.770148 30152 raft_consensus.cc:374] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:04.770179 30152 raft_consensus.cc:729] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8402d498b248649e33f9cb69ac9ceb, State: Initialized, Role: FOLLOWER
I20240506 15:42:04.770298 30152 consensus_queue.cc:260] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8402d498b248649e33f9cb69ac9ceb" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33809 } } peers { permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 } } peers { permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 } }
I20240506 15:42:04.770588 30152 ts_tablet_manager.cc:1398] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 15:42:04.770637 30000 heartbeater.cc:503] Master 127.17.240.254:39925 was elected leader, sending a full tablet report...
I20240506 15:42:04.795614 30160 raft_consensus.cc:482] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:42:04.795768 30160 raft_consensus.cc:504] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8402d498b248649e33f9cb69ac9ceb" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33809 } } peers { permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 } } peers { permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 } }
I20240506 15:42:04.798861 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.796718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:40303 (local address 127.17.240.194:50225)
0506 15:42:04.796778 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:04.796788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:04.796827 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:04.797043 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:04.797044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:04.797047 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:42:04.797274 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.797277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.797959 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.797960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:04.798617 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.798620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.798708 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.798712 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:42:04.798746 (+ 34us) client_negotiation.cc:240] Negotiation successful
0506 15:42:04.798783 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20240506 15:42:04.799351 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.796784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:50225 (local address 127.17.240.195:40303)
0506 15:42:04.796837 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:04.796839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:04.796849 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:42:04.796906 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:04.796907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:04.796917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:04.796973 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:42:04.797358 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.797885 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.798699 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.799219 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.799228 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:04.799253 (+ 25us) server_negotiation.cc:299] Negotiation successful
0506 15:42:04.799288 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240506 15:42:04.799711 30085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879aef5d4c1345b6a7a51b6f81d1bf4d" candidate_uuid: "4d8402d498b248649e33f9cb69ac9ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2136c4620b9f43dda3fb16e4c56e5597" is_pre_election: true
I20240506 15:42:04.799871 30085 raft_consensus.cc:2454] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8402d498b248649e33f9cb69ac9ceb in term 0.
I20240506 15:42:04.800099 29887 leader_election.cc:302] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2136c4620b9f43dda3fb16e4c56e5597, 4d8402d498b248649e33f9cb69ac9ceb; no voters:
I20240506 15:42:04.800365 30161 raft_consensus.cc:2781] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:42:04.800444 30161 raft_consensus.cc:482] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:42:04.800482 30161 raft_consensus.cc:3037] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:04.801692 30161 raft_consensus.cc:504] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8402d498b248649e33f9cb69ac9ceb" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33809 } } peers { permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 } } peers { permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 } }
I20240506 15:42:04.802299 30161 leader_election.cc:288] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [CANDIDATE]: Term 1 election: Requested vote from peers 2136c4620b9f43dda3fb16e4c56e5597 (127.17.240.195:40303), 7ed9dec23763460fac696147f0dfd664 (127.17.240.193:44873)
I20240506 15:42:04.802378 30085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879aef5d4c1345b6a7a51b6f81d1bf4d" candidate_uuid: "4d8402d498b248649e33f9cb69ac9ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2136c4620b9f43dda3fb16e4c56e5597"
I20240506 15:42:04.802436 30085 raft_consensus.cc:3037] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:04.803119 30085 raft_consensus.cc:2454] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8402d498b248649e33f9cb69ac9ceb in term 1.
I20240506 15:42:04.803787 29887 leader_election.cc:302] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2136c4620b9f43dda3fb16e4c56e5597, 4d8402d498b248649e33f9cb69ac9ceb; no voters:
I20240506 15:42:04.804782 30161 raft_consensus.cc:2781] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:42:04.804937 30161 raft_consensus.cc:686] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 1 LEADER]: Becoming Leader. State: Replica: 4d8402d498b248649e33f9cb69ac9ceb, State: Running, Role: LEADER
I20240506 15:42:04.805033 30161 consensus_queue.cc:237] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8402d498b248649e33f9cb69ac9ceb" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33809 } } peers { permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 } } peers { permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 } }
I20240506 15:42:04.805227 30160 leader_election.cc:288] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2136c4620b9f43dda3fb16e4c56e5597 (127.17.240.195:40303), 7ed9dec23763460fac696147f0dfd664 (127.17.240.193:44873)
I20240506 15:42:04.805790 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.802171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:44873 (local address 127.17.240.194:46949)
0506 15:42:04.802212 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:04.802222 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:04.802256 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:04.803379 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:04.803380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:04.803382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:42:04.803545 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.803547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.804767 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.804768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:04.805574 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:04.805577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.805653 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.805657 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:42:04.805690 (+ 33us) client_negotiation.cc:240] Negotiation successful
0506 15:42:04.805728 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20240506 15:42:04.806270 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.803119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:46949 (local address 127.17.240.193:44873)
0506 15:42:04.803220 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:04.803222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:04.803231 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:42:04.803258 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:04.803259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:04.803269 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:04.803320 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:42:04.804023 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.804687 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.805649 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.806134 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.806143 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:04.806167 (+ 24us) server_negotiation.cc:299] Negotiation successful
0506 15:42:04.806199 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20240506 15:42:04.806769 29816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879aef5d4c1345b6a7a51b6f81d1bf4d" candidate_uuid: "4d8402d498b248649e33f9cb69ac9ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9dec23763460fac696147f0dfd664"
I20240506 15:42:04.806807 29815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879aef5d4c1345b6a7a51b6f81d1bf4d" candidate_uuid: "4d8402d498b248649e33f9cb69ac9ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9dec23763460fac696147f0dfd664" is_pre_election: true
I20240506 15:42:04.806880 29816 raft_consensus.cc:3037] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:04.806869 29649 catalog_manager.cc:5484] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d8402d498b248649e33f9cb69ac9ceb (127.17.240.194). New cstate: current_term: 1 leader_uuid: "4d8402d498b248649e33f9cb69ac9ceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8402d498b248649e33f9cb69ac9ceb" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 33809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 } health_report { overall_health: UNKNOWN } } }
I20240506 15:42:04.807883 29816 raft_consensus.cc:2454] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8402d498b248649e33f9cb69ac9ceb in term 1.
I20240506 15:42:04.823942 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.820844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.17.240.194:33809)
0506 15:42:04.821002 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:04.821005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:04.821012 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:42:04.821056 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:04.821056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:04.821076 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:04.821141 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:42:04.821486 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.822270 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:04.823231 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:04.823404 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:04.823451 (+ 47us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:42:04.823564 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:42:04.823641 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:04.823823 (+ 182us) server_negotiation.cc:299] Negotiation successful
0506 15:42:04.823849 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20240506 15:42:04.825881 29816 raft_consensus.cc:1261] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 7ed9dec23763460fac696147f0dfd664 [term 1 FOLLOWER]: Refusing update from remote peer 4d8402d498b248649e33f9cb69ac9ceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:42:04.825960 30085 raft_consensus.cc:1261] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597 [term 1 FOLLOWER]: Refusing update from remote peer 4d8402d498b248649e33f9cb69ac9ceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:42:04.826335 30160 consensus_queue.cc:1035] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed9dec23763460fac696147f0dfd664" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:42:04.826494 30160 consensus_queue.cc:1035] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2136c4620b9f43dda3fb16e4c56e5597" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:42:04.833153 30165 mvcc.cc:205] Tried to move back new op lower bound from 7024681471283974144 to 7024681471202172928. Current Snapshot: MvccSnapshot[applied={T|T < 7024681471283974144}]
I20240506 15:42:04.834023 30167 mvcc.cc:205] Tried to move back new op lower bound from 7024681471283974144 to 7024681471202172928. Current Snapshot: MvccSnapshot[applied={T|T < 7024681471283974144}]
I20240506 15:42:04.834223 30166 mvcc.cc:205] Tried to move back new op lower bound from 7024681471283974144 to 7024681471202172928. Current Snapshot: MvccSnapshot[applied={T|T < 7024681471283974144}]
I20240506 15:42:04.842025 29650 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36088:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:42:04.842154 29650 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36088:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:42:04.843132 29650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 8b172981d9b647058c5e8eaf5cf2ec18: Sending DeleteTablet for 3 replicas of tablet 879aef5d4c1345b6a7a51b6f81d1bf4d
I20240506 15:42:04.843720 29934 tablet_service.cc:1514] Processing DeleteTablet for tablet 879aef5d4c1345b6a7a51b6f81d1bf4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:42:04 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:33494
I20240506 15:42:04.844000 30060 tablet_service.cc:1514] Processing DeleteTablet for tablet 879aef5d4c1345b6a7a51b6f81d1bf4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:42:04 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:45514
I20240506 15:42:04.844209 29794 tablet_service.cc:1514] Processing DeleteTablet for tablet 879aef5d4c1345b6a7a51b6f81d1bf4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:42:04 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:56944
I20240506 15:42:04.844998 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 29731
I20240506 15:42:04.847131 30174 tablet_replica.cc:330] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: stopping tablet replica
I20240506 15:42:04.847154 30175 tablet_replica.cc:330] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: stopping tablet replica
I20240506 15:42:04.847265 30174 raft_consensus.cc:2229] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:42:04.847381 30174 raft_consensus.cc:2258] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:42:04.847517 30175 raft_consensus.cc:2229] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:42:04.849647 30174 ts_tablet_manager.cc:1875] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:42:04.849800 30175 raft_consensus.cc:2258] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:42:04.850354 30175 ts_tablet_manager.cc:1875] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:42:04.851665 30174 ts_tablet_manager.cc:1888] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:42:04.851747 30174 log.cc:1196] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/879aef5d4c1345b6a7a51b6f81d1bf4d
I20240506 15:42:04.852078 30174 ts_tablet_manager.cc:1909] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 4d8402d498b248649e33f9cb69ac9ceb: Deleting consensus metadata
I20240506 15:42:04.852517 29636 catalog_manager.cc:4830] TS 4d8402d498b248649e33f9cb69ac9ceb (127.17.240.194:33809): tablet 879aef5d4c1345b6a7a51b6f81d1bf4d (table test-table [id=8692a7027fa940a99f98ffc6f75e73c8]) successfully deleted
I20240506 15:42:04.853022 30175 ts_tablet_manager.cc:1888] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:42:04.853098 30175 log.cc:1196] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/879aef5d4c1345b6a7a51b6f81d1bf4d
I20240506 15:42:04.853386 30175 ts_tablet_manager.cc:1909] T 879aef5d4c1345b6a7a51b6f81d1bf4d P 2136c4620b9f43dda3fb16e4c56e5597: Deleting consensus metadata
W20240506 15:42:04.856251 29634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:44873 (error 108)
I20240506 15:42:04.857159 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:04.857036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:44873 (local address 127.0.0.1:56958)
0506 15:42:04.857091 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:04.857120 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20240506 15:42:04.857370 29634 catalog_manager.cc:4573] TS 7ed9dec23763460fac696147f0dfd664 (127.17.240.193:44873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 879aef5d4c1345b6a7a51b6f81d1bf4d: Network error: Client connection negotiation failed: client connection to 127.17.240.193:44873: connect: Connection refused (error 111)
I20240506 15:42:04.857636 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 29870
I20240506 15:42:04.859555 29634 catalog_manager.cc:4830] TS 2136c4620b9f43dda3fb16e4c56e5597 (127.17.240.195:40303): tablet 879aef5d4c1345b6a7a51b6f81d1bf4d (table test-table [id=8692a7027fa940a99f98ffc6f75e73c8]) successfully deleted
I20240506 15:42:04.864017 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 30005
I20240506 15:42:04.869293 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 29618
I20240506 15:42:04.874563 18371 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240506 15:42:05.219591 18371 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240506 15:42:05.219736 18371 mini_ranger.cc:66] Stopping Ranger...
I20240506 15:42:05.564888 18371 mini_ranger.cc:68] Stopped Ranger
2024-05-06 15:42:05.565 UTC [29204] LOG: received smart shutdown request
2024-05-06 15:42:05.569 UTC [29204] LOG: background worker "logical replication launcher" (PID 29213) exited with exit code 1
2024-05-06 15:42:05.569 UTC [29208] LOG: shutting down
2024-05-06 15:42:05.759 UTC [29204] LOG: database system is shut down
2024-05-06T15:42:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46996 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:42:05 dist-test-slave-v0k1 krb5kdc[30186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:42:05 dist-test-slave-v0k1 krb5kdc[30186](info): set up 2 sockets
May 06 15:42:05 dist-test-slave-v0k1 krb5kdc[30186](info): commencing operation
krb5kdc: starting...
W20240506 15:42:06.835170 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:42:06 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:42:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:42:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/rangeradmin_127.17.240.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/rangeradmin_127.17.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/rangerlookup_127.17.240.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/rangerlookup_127.17.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/HTTP_127.17.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/HTTP_127.17.240.212@KRBTEST.COM.keytab.
I20240506 15:42:07.001502 18371 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskb5AjvH/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/postgres -l logfile start
2024-05-06 15:42:09.957 UTC [30216] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-06 15:42:09.957 UTC [30216] LOG: listening on IPv4 address "127.17.240.212", port 37467
2024-05-06 15:42:09.961 UTC [30216] LOG: listening on Unix socket "/tmp/.s.PGSQL.37467"
2024-05-06 15:42:09.982 UTC [30219] LOG: database system was shut down at 2024-05-06 15:42:08 UTC
2024-05-06 15:42:09.986 UTC [30216] LOG: database system is ready to accept connections
I20240506 15:42:10.871224 18371 mini_postgres.cc:96] Postgres bound to 37467
2024-05-06 15:42:10.876 UTC [30227] FATAL: database "slave" does not exist
127.17.240.212:37467 - accepting connections
I20240506 15:42:10.877378 18371 mini_ranger.cc:161] Starting Ranger...
I20240506 15:42:10.891507 18371 mini_ranger.cc:84] Created miniranger Postgres user
I20240506 15:42:11.476275 18371 mini_ranger.cc:87] Created ranger Postgres database
I20240506 15:42:11.476362 18371 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-admin
2024-05-06 15:42:11,679 [I] DB FLAVOR :POSTGRES
2024-05-06 15:42:11,679 [I] --------- Verifying Ranger DB connection ---------
2024-05-06 15:42:11,679 [I] Checking connection..
2024-05-06 15:42:11,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-06 15:42:11,977 [I] Checking connection passed.
2024-05-06 15:42:11,977 [I] --------- Verifying version history table ---------
2024-05-06 15:42:11,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-06 15:42:12,221 [I] Table x_db_version_h does not exist in database ranger
2024-05-06 15:42:12,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-06 15:42:12,509 [I] Table x_db_version_h does not exist in database ranger
2024-05-06 15:42:12,510 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-06 15:42:12,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-06 15:42:12.765 UTC [30291] WARNING: there is no transaction in progress
2024-05-06 15:42:12,772 [I] create_dbversion_catalog.sql file imported successfully
2024-05-06 15:42:12,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-06 15:42:13,020 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-06 15:42:13,021 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-06 15:42:13,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-06 15:42:13,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-06 15:42:13,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v0k1.c.gcp-upstream.internal','N') ;"
2024-05-06 15:42:13,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-06 15:42:13,981 [I] Table x_portal_user does not exist in database ranger
2024-05-06 15:42:13,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-06 15:42:14,227 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-06 15:42:14,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-06 15:42:14,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-06 15:42:14,725 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-06 15:42:14,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-06 15:42:15.782 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:15.801 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:15.818 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:15.835 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:15.856 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:15.878 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:15.947 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:15.955 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:15.970 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:15.981 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:15.991 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:16.005 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:16.015 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:16.023 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:16.030 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:16.038 UTC [30399] WARNING: there is no transaction in progress
2024-05-06 15:42:16,824 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-06 15:42:16,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-06 15:42:17,079 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-06 15:42:17,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-06 15:42:17,338 [I] Table x_portal_user already exists in database 'ranger'
2024-05-06 15:42:17,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-06 15:42:17,604 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-06 15:42:17,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-06 15:42:17,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-06 15:42:18,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v0k1.c.gcp-upstream.internal';"
2024-05-06 15:42:18,367 [I] CORE_DB_SCHEMA import status has been updated
2024-05-06 15:42:18,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-06 15:42:18,623 [I] DB_PATCHES have already been applied
I20240506 15:42:18.629163 18371 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-admin:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskb5AjvH/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240506 15:42:18.629242 18371 mini_ranger.cc:193] Using host: 127.17.240.212
I20240506 15:42:18.632097 18371 mini_ranger.cc:227] Ranger admin URL: http://127.17.240.212:54205
May 06, 2024 3:42:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 06, 2024 3:42:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 06, 2024 3:42:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-admin/ews/webapp
May 06, 2024 3:42:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 06, 2024 3:42:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-admin/ews/webapp] .....
May 06, 2024 3:42:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-admin/ews/webapp].
May 06, 2024 3:42:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240506 15:42:19.649209 18371 mini_ranger.cc:160] Time spent starting Ranger: real 8.772s user 0.000s sys 0.009s
May 06, 2024 3:42:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 06, 2024 3:42:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
May 06, 2024 3:42:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 06, 2024 3:42:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 06, 2024 3:42:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240506 15:42:40.834527 18371 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/rangerkms_127.17.240.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/rangerkms_127.17.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.240.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.240.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.240.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/HTTP_127.17.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.240.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/HTTP_127.17.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 06 15:42:40 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010160, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240506 15:42:40.920704 18371 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240506 15:42:40.930307 18371 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240506 15:42:41.935421 18371 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240506 15:42:41.935513 18371 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-kms
I20240506 15:42:41.935536 18371 mini_ranger_kms.cc:227] Using postgres at 127.17.240.212:37467
2024-05-06 15:42:42,101 [I] DB FLAVOR :POSTGRES
2024-05-06 15:42:42,101 [I] --------- Verifying Ranger DB connection ---------
2024-05-06 15:42:42,101 [I] Checking connection
2024-05-06 15:42:42,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-06 15:42:42,365 [I] connection success
2024-05-06 15:42:42,365 [I] --------- Verifying Ranger DB tables ---------
2024-05-06 15:42:42,366 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-06 15:42:42,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-06 15:42:42,630 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-06 15:42:42,630 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-06 15:42:42,630 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-06 15:42:42,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.240.212:37467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-06 15:42:42,906 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240506 15:42:43.404440 18371 mini_ranger_kms.cc:310] Created kms service
I20240506 15:42:43.595471 18371 mini_ranger_kms.cc:326] Created kudu user
I20240506 15:42:43.641954 18371 mini_ranger_kms.cc:343] Created rangerkms user
May 06 15:42:43 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010160, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:44.037245 18371 mini_ranger_kms.cc:384] Added ranger policy
I20240506 15:42:44.037451 18371 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-kms:/tmp/dist-test-taskb5AjvH/build/release/bin/postgresql.jar:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskb5AjvH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240506 15:42:44.037474 18371 mini_ranger_kms.cc:240] Using host: 127.17.240.212
I20240506 15:42:44.040642 18371 mini_ranger_kms.cc:276] Ranger KMS PID: 30579
I20240506 15:42:44.040726 18371 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.240.212:39089
May 06, 2024 3:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 06, 2024 3:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 06, 2024 3:42:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 06, 2024 3:42:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-kms/ews/webapp] .....
May 06, 2024 3:42:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-kms/ews/webapp].
May 06, 2024 3:42:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240506 15:42:46.123157 18371 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.202s user 0.000s sys 0.148s
I20240506 15:42:46.123262 18371 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240506 15:42:46.123309 18371 mini_ranger_kms.cc:401] 127.17.240.212:39089/kms/v1/keys
May 06 15:42:46 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1715010166, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.240.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 06 15:42:46 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010166, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.240.212@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
May 06 15:42:47 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010166, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.240.212@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
May 06 15:42:49 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:42:49.469944 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:35763
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:35763
--ranger_config_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:39089/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:49.548663 30630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:49.548859 30630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:49.548892 30630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:49.550125 30630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:42:49.550165 30630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:49.550189 30630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:49.550209 30630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:42:49.550246 30630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:42:49.551862 30630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.240.212:39089/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:35763
--ranger_config_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:35763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:49.552064 30630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:49.552232 30630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:49.557353 30638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:49.558045 30635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:49.558667 30636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:49.559233 30630 server_base.cc:1034] running on GCE node
I20240506 15:42:49.559448 30630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:49.559692 30630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:49.560832 30630 hybrid_clock.cc:648] HybridClock initialized: now 1715010169560815 us; error 35 us; skew 500 ppm
May 06 15:42:49 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010169, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:49.567880 30630 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:42:49.568351 30630 webserver.cc:460] Webserver started at http://127.17.240.254:40751/ using document root <none> and password file <none>
I20240506 15:42:49.568547 30630 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:49.568583 30630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:49.568693 30630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 06 15:42:49 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010169, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:49.628304 30630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "35a681f18ea244ccab3258c090f60d21"
format_stamp: "Formatted at 2024-05-06 15:42:49 on dist-test-slave-v0k1"
server_key: "0818a198c0cf8bc2487aacd605f32ead"
server_key_iv: "9aab58c5344a59c0e021ee04c2a7f281"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:49.628676 30630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "35a681f18ea244ccab3258c090f60d21"
format_stamp: "Formatted at 2024-05-06 15:42:49 on dist-test-slave-v0k1"
server_key: "0818a198c0cf8bc2487aacd605f32ead"
server_key_iv: "9aab58c5344a59c0e021ee04c2a7f281"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:49.629943 30630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:42:49.644757 30645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:49.645031 30630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240506 15:42:49.645126 30630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "35a681f18ea244ccab3258c090f60d21"
format_stamp: "Formatted at 2024-05-06 15:42:49 on dist-test-slave-v0k1"
server_key: "0818a198c0cf8bc2487aacd605f32ead"
server_key_iv: "9aab58c5344a59c0e021ee04c2a7f281"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:49.645220 30630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:49.655297 30630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:49.657356 30630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:49.657488 30630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:49.662998 30630 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:35763
I20240506 15:42:49.663076 30697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:35763 every 8 connection(s)
I20240506 15:42:49.663425 30630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:42:49.664146 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:49.665311 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 30630
I20240506 15:42:49.665457 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:42:49.665654 18371 external_mini_cluster.cc:1428] Setting key 22328bb2eae5a1e8625086fc2fd90487
I20240506 15:42:49.666990 30698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: Bootstrap starting.
I20240506 15:42:49.667670 30698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:49.667960 30698 log.cc:824] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:49.668682 30698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: No bootstrap required, opened a new log
I20240506 15:42:49.671201 30698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } }
I20240506 15:42:49.671380 30698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:49.671409 30698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a681f18ea244ccab3258c090f60d21, State: Initialized, Role: FOLLOWER
I20240506 15:42:49.671572 30698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } }
I20240506 15:42:49.671739 30698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:42:49.671815 30698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:42:49.671893 30698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:49.672605 30698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } }
I20240506 15:42:49.672717 30698 leader_election.cc:302] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35a681f18ea244ccab3258c090f60d21; no voters:
I20240506 15:42:49.672921 30698 leader_election.cc:288] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:42:49.672966 30703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:42:49.673185 30703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 1 LEADER]: Becoming Leader. State: Replica: 35a681f18ea244ccab3258c090f60d21, State: Running, Role: LEADER
I20240506 15:42:49.673236 30698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:42:49.673305 30703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } }
I20240506 15:42:49.673668 30703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35a681f18ea244ccab3258c090f60d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } } }
I20240506 15:42:49.673831 30703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:49.674006 30703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35a681f18ea244ccab3258c090f60d21. Latest consensus state: current_term: 1 leader_uuid: "35a681f18ea244ccab3258c090f60d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } } }
I20240506 15:42:49.674106 30703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:49.674247 30711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:42:49.674772 30711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:42:49.676031 30719 mvcc.cc:205] Tried to move back new op lower bound from 7024681654991196160 to 7024681654982361088. Current Snapshot: MvccSnapshot[applied={T|T < 7024681654991196160}]
May 06 15:42:49 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:49.677454 30698 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240506 15:42:49.682063 30711 catalog_manager.cc:1347] Generated new cluster ID: d7615f6952c04714a7de9479f71a2d9c
I20240506 15:42:49.682164 30711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:42:49.690781 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:49.666134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.17.240.254:35763)
0506 15:42:49.666297 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:49.666306 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:49.671270 (+ 4964us) server_negotiation.cc:407] Connection header received
0506 15:42:49.671428 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:49.671434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:49.671490 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:49.671628 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:49.672495 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:49.673597 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:49.674684 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:49.674925 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:49.677219 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:49.677227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:49.677227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:49.677244 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:49.688723 (+ 11479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:49.689209 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:49.689213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:49.689213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:49.689276 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:49.689460 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:49.689461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:49.689461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:49.689728 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:49.690331 (+ 603us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:49.690415 (+ 84us) server_negotiation.cc:299] Negotiation successful
0506 15:42:49.690561 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20240506 15:42:49.697824 30711 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:42:49.698673 30711 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:42:49.708060 30711 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: Generated new TSK 0
I20240506 15:42:49.708432 30711 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:42:50.463351 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:39089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35763
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:50.579473 30743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:50.579730 30743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:50.579762 30743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:50.581831 30743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:50.581969 30743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:50.582049 30743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:42:50.584676 30743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.240.212:39089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:50.585175 30743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:50.585446 30743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:50.591194 30749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:50.591817 30748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:50.592360 30751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:50.592917 30743 server_base.cc:1034] running on GCE node
I20240506 15:42:50.593207 30743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:50.593448 30743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:50.594605 30743 hybrid_clock.cc:648] HybridClock initialized: now 1715010170594577 us; error 48 us; skew 500 ppm
May 06 15:42:50 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010170, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:50.597813 30743 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:42:50.598287 30743 webserver.cc:460] Webserver started at http://127.17.240.193:46143/ using document root <none> and password file <none>
I20240506 15:42:50.598481 30743 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:50.598520 30743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:50.598614 30743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 06 15:42:50 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010170, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:50.615414 30743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "0177154837644b34be29eee40aaea3ab"
format_stamp: "Formatted at 2024-05-06 15:42:50 on dist-test-slave-v0k1"
server_key: "aa567ff9cfab480cdb71faa3c67d01af"
server_key_iv: "a2f6dc8f8aeada3b9a208a7892cc8810"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:50.615834 30743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "0177154837644b34be29eee40aaea3ab"
format_stamp: "Formatted at 2024-05-06 15:42:50 on dist-test-slave-v0k1"
server_key: "aa567ff9cfab480cdb71faa3c67d01af"
server_key_iv: "a2f6dc8f8aeada3b9a208a7892cc8810"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:50.617523 30743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:42:50.631160 30758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:50.632611 30743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20240506 15:42:50.632786 30743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "0177154837644b34be29eee40aaea3ab"
format_stamp: "Formatted at 2024-05-06 15:42:50 on dist-test-slave-v0k1"
server_key: "aa567ff9cfab480cdb71faa3c67d01af"
server_key_iv: "a2f6dc8f8aeada3b9a208a7892cc8810"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:50.632952 30743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:50.658771 30743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:50.661031 30743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:50.661232 30743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:50.661477 30743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:42:50.661885 30743 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:42:50.661975 30743 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:50.662078 30743 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:42:50.662163 30743 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:50.669399 30743 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:44383
I20240506 15:42:50.669924 30743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:42:50.674052 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 30743
I20240506 15:42:50.674175 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:42:50.674404 18371 external_mini_cluster.cc:1428] Setting key 807c55d3e5816226f15bd089ec572b85
I20240506 15:42:50.679612 30873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:44383 every 8 connection(s)
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
May 06 15:42:50 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010170, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:50.702416 30877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:50.683436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:35763 (local address 127.17.240.193:46685)
0506 15:42:50.690787 (+ 7351us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:50.690828 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:50.691529 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:50.691826 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:50.691831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:50.692093 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:50.692545 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:50.692551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:50.693700 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:50.693701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:50.694347 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:50.694349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:50.694484 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:50.697477 (+ 2993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:50.697506 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:50.699080 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:50.701062 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:50.701068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:50.701071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:50.701330 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:50.701485 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:50.701486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:50.701486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:50.701556 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:50.701789 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:50.701792 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:50.701955 (+ 163us) client_negotiation.cc:769] Sending connection context
0506 15:42:50.702063 (+ 108us) client_negotiation.cc:240] Negotiation successful
0506 15:42:50.702207 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7079,"thread_start_us":129,"threads_started":1}
I20240506 15:42:50.702973 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:50.683784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:46685 (local address 127.17.240.254:35763)
0506 15:42:50.690469 (+ 6685us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:50.690471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:50.691442 (+ 971us) server_negotiation.cc:407] Connection header received
0506 15:42:50.691654 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:50.691656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:50.691673 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:50.691761 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:50.692664 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:50.693598 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:50.695057 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:50.695248 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:50.699187 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:50.699192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:50.699193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:50.699200 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:50.700972 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:50.701402 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:50.701403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:50.701403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:50.701427 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:50.701610 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:50.701611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:50.701611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:50.701718 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:50.701768 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:50.702763 (+ 995us) server_negotiation.cc:299] Negotiation successful
0506 15:42:50.702859 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6645,"thread_start_us":98,"threads_started":1}
I20240506 15:42:50.703807 30874 heartbeater.cc:347] Connected to a master server at 127.17.240.254:35763
I20240506 15:42:50.703928 30874 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:50.704169 30874 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
I20240506 15:42:50.704771 30662 ts_manager.cc:194] Registered new tserver with Master: 0177154837644b34be29eee40aaea3ab (127.17.240.193:44383)
I20240506 15:42:50.705750 30662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:46685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:42:50.738943 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:39089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35763
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 06 15:42:50 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1715010170, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20240506 15:42:50.860394 30882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:50.860685 30882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:50.860728 30882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:50.863199 30882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:50.863330 30882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:50.863406 30882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:42:50.866159 30882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.240.212:39089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:50.866581 30882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:50.866860 30882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:50.872992 30888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:50.873499 30887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:50.873991 30890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:50.875367 30882 server_base.cc:1034] running on GCE node
I20240506 15:42:50.875602 30882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:50.875854 30882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:50.877017 30882 hybrid_clock.cc:648] HybridClock initialized: now 1715010170876985 us; error 48 us; skew 500 ppm
May 06 15:42:50 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010170, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:50.881121 30882 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:42:50.881659 30882 webserver.cc:460] Webserver started at http://127.17.240.194:33659/ using document root <none> and password file <none>
I20240506 15:42:50.881953 30882 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:50.882045 30882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:50.882189 30882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 06 15:42:50 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010170, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:50.896014 30882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "fa361494b6b44c098200535727c6afd2"
format_stamp: "Formatted at 2024-05-06 15:42:50 on dist-test-slave-v0k1"
server_key: "a61e635d69dad8866180a60698e5a75b"
server_key_iv: "758f6f0224a76d27fd73afe70e84849b"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:50.896546 30882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "fa361494b6b44c098200535727c6afd2"
format_stamp: "Formatted at 2024-05-06 15:42:50 on dist-test-slave-v0k1"
server_key: "a61e635d69dad8866180a60698e5a75b"
server_key_iv: "758f6f0224a76d27fd73afe70e84849b"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:50.898303 30882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:42:50.911043 30897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:50.911350 30882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:42:50.911454 30882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "fa361494b6b44c098200535727c6afd2"
format_stamp: "Formatted at 2024-05-06 15:42:50 on dist-test-slave-v0k1"
server_key: "a61e635d69dad8866180a60698e5a75b"
server_key_iv: "758f6f0224a76d27fd73afe70e84849b"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:50.911548 30882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:50.930181 30882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:50.932304 30882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:50.932436 30882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:50.932682 30882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:42:50.933066 30882 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:42:50.933116 30882 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:50.933156 30882 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:42:50.933179 30882 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:50.940272 30882 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:46413
I20240506 15:42:50.940339 31011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:46413 every 8 connection(s)
I20240506 15:42:50.940625 30882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
May 06 15:42:50 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010170, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:50.949133 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:50.941548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:39805 (local address 127.17.240.254:35763)
0506 15:42:50.941592 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:50.941594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:50.941604 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:42:50.942613 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:50.942614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:50.942634 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:50.942688 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:50.943390 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:50.943947 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:50.944483 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:50.944677 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:50.946386 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:50.946390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:50.946391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:50.946397 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:50.947899 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:50.948257 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:50.948257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:50.948258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:50.948282 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:50.948452 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:50.948453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:50.948453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:50.948560 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:50.948627 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:50.948933 (+ 306us) server_negotiation.cc:299] Negotiation successful
0506 15:42:50.949033 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240506 15:42:50.949270 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:50.941316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:35763 (local address 127.17.240.194:39805)
0506 15:42:50.941483 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:50.941516 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:50.942490 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:50.942750 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:50.942753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:50.942941 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:50.943295 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:50.943299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:50.944038 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:50.944039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:50.944404 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:50.944405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:50.944535 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:50.945112 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:50.945128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:50.946294 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:50.947981 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:50.947986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:50.947989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:50.948192 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:50.948336 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:50.948337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:50.948337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:50.948397 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:50.948624 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:50.948629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:50.948808 (+ 179us) client_negotiation.cc:769] Sending connection context
0506 15:42:50.948910 (+ 102us) client_negotiation.cc:240] Negotiation successful
0506 15:42:50.949031 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20240506 15:42:50.949694 31012 heartbeater.cc:347] Connected to a master server at 127.17.240.254:35763
I20240506 15:42:50.949798 31012 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:50.950002 31012 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
I20240506 15:42:50.950398 30662 ts_manager.cc:194] Registered new tserver with Master: fa361494b6b44c098200535727c6afd2 (127.17.240.194:46413)
I20240506 15:42:50.950428 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 30882
I20240506 15:42:50.950553 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:42:50.950742 18371 external_mini_cluster.cc:1428] Setting key 8c34497743f0f2ac4baa8c2cb2cf8d71
I20240506 15:42:50.950805 30662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:39805
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:42:50.976032 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:39089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35763
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:51.094812 31017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:51.095105 31017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:51.095156 31017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:51.097265 31017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:51.097326 31017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:51.097368 31017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:42:51.099809 31017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.240.212:39089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:35763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:51.100059 31017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:51.100282 31017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:51.110216 31023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:51.110684 31025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:51.111289 31022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:51.112013 31017 server_base.cc:1034] running on GCE node
I20240506 15:42:51.112217 31017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:51.112470 31017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:51.113631 31017 hybrid_clock.cc:648] HybridClock initialized: now 1715010171113618 us; error 29 us; skew 500 ppm
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:51.116245 31017 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:42:51.116690 31017 webserver.cc:460] Webserver started at http://127.17.240.195:42619/ using document root <none> and password file <none>
I20240506 15:42:51.116874 31017 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:51.116914 31017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:51.117022 31017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:51.143790 31017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "2f47e7bb6995420e90bb837433dd22a0"
format_stamp: "Formatted at 2024-05-06 15:42:51 on dist-test-slave-v0k1"
server_key: "ac41554ac27a3eb9526cca2b6408a917"
server_key_iv: "5573c208f031b23da0d170187ab139d6"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:51.144547 31017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "2f47e7bb6995420e90bb837433dd22a0"
format_stamp: "Formatted at 2024-05-06 15:42:51 on dist-test-slave-v0k1"
server_key: "ac41554ac27a3eb9526cca2b6408a917"
server_key_iv: "5573c208f031b23da0d170187ab139d6"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:51.151453 31017 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20240506 15:42:51.171198 31032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:51.177487 31017 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20240506 15:42:51.177690 31017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "2f47e7bb6995420e90bb837433dd22a0"
format_stamp: "Formatted at 2024-05-06 15:42:51 on dist-test-slave-v0k1"
server_key: "ac41554ac27a3eb9526cca2b6408a917"
server_key_iv: "5573c208f031b23da0d170187ab139d6"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:51.177865 31017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:51.201453 31017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:51.204314 31017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:51.204520 31017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:51.204789 31017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:42:51.205154 31017 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:42:51.205226 31017 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:51.205288 31017 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:42:51.205336 31017 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:51.213305 31017 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:40539
I20240506 15:42:51.213725 31017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:42:51.213981 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 31017
I20240506 15:42:51.214083 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:42:51.214279 18371 external_mini_cluster.cc:1428] Setting key 866b7f60e85014937846e0014e22833d
I20240506 15:42:51.214736 31145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:40539 every 8 connection(s)
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:51.225379 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:51.215642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:58307 (local address 127.17.240.254:35763)
0506 15:42:51.215892 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:51.215894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:51.216085 (+ 191us) server_negotiation.cc:407] Connection header received
0506 15:42:51.217141 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:51.217142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:51.217155 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:51.217228 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:51.218193 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.218970 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:51.219806 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.219983 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:51.222097 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:51.222102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:51.222102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:51.222109 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:51.223964 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:51.224418 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:51.224419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:51.224420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:51.224445 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:51.224654 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:51.224655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:51.224655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:51.224764 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:51.224805 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:51.225182 (+ 377us) server_negotiation.cc:299] Negotiation successful
0506 15:42:51.225276 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20240506 15:42:51.225865 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:51.215663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:35763 (local address 127.17.240.195:58307)
0506 15:42:51.215953 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:51.215988 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:51.217010 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:51.217290 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:51.217295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:51.217583 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:51.218047 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:51.218053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.219129 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:51.219130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:51.219701 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:51.219703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.219870 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:51.220560 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:51.220583 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:51.221980 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:51.224058 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:51.224064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:51.224067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:51.224330 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:51.224510 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:51.224511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:51.224511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:51.224590 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:51.224848 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:51.224853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:51.225017 (+ 164us) client_negotiation.cc:769] Sending connection context
0506 15:42:51.225535 (+ 518us) client_negotiation.cc:240] Negotiation successful
0506 15:42:51.225668 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":97,"threads_started":1}
I20240506 15:42:51.226294 31146 heartbeater.cc:347] Connected to a master server at 127.17.240.254:35763
I20240506 15:42:51.226397 31146 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:51.226617 31146 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
I20240506 15:42:51.227073 30662 ts_manager.cc:194] Registered new tserver with Master: 2f47e7bb6995420e90bb837433dd22a0 (127.17.240.195:40539)
I20240506 15:42:51.227591 30662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:58307
I20240506 15:42:51.237253 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:42:51.237991 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 30743
I20240506 15:42:51.243745 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 30882
I20240506 15:42:51.249399 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 31017
I20240506 15:42:51.255858 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 30630
I20240506 15:42:51.261440 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:35763
--webserver_interface=127.17.240.254
--webserver_port=40751
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:35763
--ranger_config_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:39089/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.240.212:39089/kms with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:51.341864 31149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:51.342249 31149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:51.342272 31149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:51.343578 31149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:42:51.343621 31149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:51.343634 31149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:51.343647 31149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:42:51.343668 31149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:42:51.345227 31149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.240.212:39089/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:35763
--ranger_config_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:35763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:51.345445 31149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:51.345629 31149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:51.350245 31154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:51.350767 31155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:51.351271 31157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:51.352059 31149 server_base.cc:1034] running on GCE node
I20240506 15:42:51.352281 31149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:51.352540 31149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:51.353674 31149 hybrid_clock.cc:648] HybridClock initialized: now 1715010171353654 us; error 34 us; skew 500 ppm
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:51.356058 31149 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:42:51.356452 31149 webserver.cc:460] Webserver started at http://127.17.240.254:40751/ using document root <none> and password file <none>
I20240506 15:42:51.356634 31149 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:51.356688 31149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:51.404179 31149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:42:51.405054 31165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:51.405215 31149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:42:51.405295 31149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "35a681f18ea244ccab3258c090f60d21"
format_stamp: "Formatted at 2024-05-06 15:42:49 on dist-test-slave-v0k1"
server_key: "0818a198c0cf8bc2487aacd605f32ead"
server_key_iv: "9aab58c5344a59c0e021ee04c2a7f281"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:51.405676 31149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:51.431824 31149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:51.433532 31149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:51.433646 31149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:51.437530 31149 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:35763
I20240506 15:42:51.437602 31217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:35763 every 8 connection(s)
I20240506 15:42:51.437881 31149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:42:51.439033 31218 sys_catalog.cc:263] Verifying existing consensus state
I20240506 15:42:51.439604 31218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: Bootstrap starting.
I20240506 15:42:51.446983 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 31149
I20240506 15:42:51.447351 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:39089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:44383
--local_ip_for_outbound_sockets=127.17.240.193
--tserver_master_addrs=127.17.240.254:35763
--webserver_port=46143
--webserver_interface=127.17.240.193
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.240.212:39089/kms with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:42:51.448611 31218 log.cc:824] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:51.450217 31218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:42:51.450495 31218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: Bootstrap complete.
I20240506 15:42:51.452808 31218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } }
I20240506 15:42:51.453018 31218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a681f18ea244ccab3258c090f60d21, State: Initialized, Role: FOLLOWER
I20240506 15:42:51.453161 31218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } }
I20240506 15:42:51.453239 31218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:42:51.453281 31218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:42:51.453336 31218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:42:51.454253 31218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } }
I20240506 15:42:51.454363 31218 leader_election.cc:302] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35a681f18ea244ccab3258c090f60d21; no voters:
I20240506 15:42:51.454533 31218 leader_election.cc:288] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240506 15:42:51.454582 31222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:42:51.454722 31222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [term 2 LEADER]: Becoming Leader. State: Replica: 35a681f18ea244ccab3258c090f60d21, State: Running, Role: LEADER
I20240506 15:42:51.454798 31218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:42:51.454803 31222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } }
I20240506 15:42:51.455114 31222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35a681f18ea244ccab3258c090f60d21. Latest consensus state: current_term: 2 leader_uuid: "35a681f18ea244ccab3258c090f60d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } } }
I20240506 15:42:51.455200 31222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:51.455121 31224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "35a681f18ea244ccab3258c090f60d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a681f18ea244ccab3258c090f60d21" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 35763 } } }
I20240506 15:42:51.455382 31224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21 [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:51.455838 31226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:42:51.456423 31226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:42:51.456607 31226 catalog_manager.cc:1259] Loaded cluster ID: d7615f6952c04714a7de9479f71a2d9c
I20240506 15:42:51.456692 31226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:42:51.456760 31218 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240506 15:42:51.458271 31226 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:42:51.458456 31226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: Loaded TSK: 0
I20240506 15:42:51.458678 31226 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240506 15:42:51.554724 31220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:51.554989 31220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:51.555054 31220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:51.557252 31220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:51.557318 31220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:51.557363 31220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:42:51.559751 31220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.240.212:39089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:44383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=46143
--tserver_master_addrs=127.17.240.254:35763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:51.559993 31220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:51.560194 31220 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:51.565744 31258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:51.566429 31255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:51.566977 31256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:51.567515 31220 server_base.cc:1034] running on GCE node
I20240506 15:42:51.567720 31220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:51.567994 31220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:51.569176 31220 hybrid_clock.cc:648] HybridClock initialized: now 1715010171569135 us; error 59 us; skew 500 ppm
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:51.572551 31220 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:42:51.573101 31220 webserver.cc:460] Webserver started at http://127.17.240.193:46143/ using document root <none> and password file <none>
I20240506 15:42:51.573333 31220 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:51.573400 31220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:51.599503 31220 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240506 15:42:51.600252 31266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:51.600450 31220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240506 15:42:51.600523 31220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "0177154837644b34be29eee40aaea3ab"
format_stamp: "Formatted at 2024-05-06 15:42:50 on dist-test-slave-v0k1"
server_key: "aa567ff9cfab480cdb71faa3c67d01af"
server_key_iv: "a2f6dc8f8aeada3b9a208a7892cc8810"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:51.600924 31220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:51.625084 31220 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:51.627693 31220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:51.627890 31220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:51.628130 31220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:42:51.628530 31220 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:42:51.628609 31220 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:51.628674 31220 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:42:51.628720 31220 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:51.635378 31220 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:44383
I20240506 15:42:51.635859 31220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:42:51.638535 31379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:44383 every 8 connection(s)
I20240506 15:42:51.644718 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 31220
I20240506 15:42:51.645058 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:39089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:46413
--local_ip_for_outbound_sockets=127.17.240.194
--tserver_master_addrs=127.17.240.254:35763
--webserver_port=33659
--webserver_interface=127.17.240.194
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.240.212:39089/kms with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:51.655934 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:51.639276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:35763 (local address 127.17.240.193:47645)
0506 15:42:51.639799 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:51.639832 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:51.640923 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:51.641575 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:51.641580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:51.641840 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:51.642275 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:51.642281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.644190 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:51.644191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:51.647111 (+ 2920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:51.647115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.647256 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:51.647946 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:51.647970 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:51.649625 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:51.654104 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:51.654113 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:51.654116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:51.654487 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:51.654736 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:51.654737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:51.654737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:51.654816 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:51.655324 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:51.655330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:51.655478 (+ 148us) client_negotiation.cc:769] Sending connection context
0506 15:42:51.655584 (+ 106us) client_negotiation.cc:240] Negotiation successful
0506 15:42:51.655722 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":90,"threads_started":1}
I20240506 15:42:51.656576 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:51.639589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:47645 (local address 127.17.240.254:35763)
0506 15:42:51.641161 (+ 1572us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:51.641173 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:51.641199 (+ 26us) server_negotiation.cc:407] Connection header received
0506 15:42:51.641327 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:51.641332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:51.641385 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:51.642409 (+ 1024us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:51.642797 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.644074 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:51.648637 (+ 4563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.648889 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:51.649742 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:51.649774 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:51.649774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:51.649794 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:51.653968 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:51.654593 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:51.654596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:51.654597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:51.654656 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:51.654892 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:51.654893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:51.654894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:51.655167 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:51.656124 (+ 957us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:51.656214 (+ 90us) server_negotiation.cc:299] Negotiation successful
0506 15:42:51.656373 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1525,"thread_start_us":95,"threads_started":1}
I20240506 15:42:51.657245 31380 heartbeater.cc:347] Connected to a master server at 127.17.240.254:35763
I20240506 15:42:51.657347 31380 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:51.657567 31380 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
W20240506 15:42:51.658425 31380 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:35763 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240506 15:42:51.658494 31380 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:51.658586 31380 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
I20240506 15:42:51.658995 31380 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:51.659129 31380 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
W20240506 15:42:51.659478 31380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240506 15:42:51.782256 31384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:51.782534 31384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:51.782575 31384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:51.784823 31384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:51.784905 31384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:51.784947 31384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:42:51.787567 31384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.240.212:39089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:46413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=33659
--tserver_master_addrs=127.17.240.254:35763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:51.787847 31384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:51.788080 31384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:51.794507 31390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:51.794948 31389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:51.795729 31392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:51.796466 31384 server_base.cc:1034] running on GCE node
I20240506 15:42:51.796660 31384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:51.796905 31384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:51.800205 31384 hybrid_clock.cc:648] HybridClock initialized: now 1715010171800176 us; error 50 us; skew 500 ppm
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:51.803481 31384 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:42:51.803979 31384 webserver.cc:460] Webserver started at http://127.17.240.194:33659/ using document root <none> and password file <none>
I20240506 15:42:51.804174 31384 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:51.804226 31384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:51.826444 31384 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240506 15:42:51.827200 31400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:51.827776 31384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240506 15:42:51.827919 31384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "fa361494b6b44c098200535727c6afd2"
format_stamp: "Formatted at 2024-05-06 15:42:50 on dist-test-slave-v0k1"
server_key: "a61e635d69dad8866180a60698e5a75b"
server_key_iv: "758f6f0224a76d27fd73afe70e84849b"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:51.828428 31384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:51.844832 31384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:51.847681 31384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:51.847882 31384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:51.848183 31384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:42:51.848587 31384 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:42:51.848677 31384 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:51.848760 31384 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:42:51.848816 31384 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:51.855832 31384 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:46413
I20240506 15:42:51.856412 31384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:42:51.859652 31513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:46413 every 8 connection(s)
I20240506 15:42:51.864964 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 31384
I20240506 15:42:51.865298 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.240.212:39089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:40539
--local_ip_for_outbound_sockets=127.17.240.195
--tserver_master_addrs=127.17.240.254:35763
--webserver_port=42619
--webserver_interface=127.17.240.195
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.240.212:39089/kms with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 06 15:42:51 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010171, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:51.873071 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:51.860494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:43337 (local address 127.17.240.254:35763)
0506 15:42:51.860547 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:51.860550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:51.860841 (+ 291us) server_negotiation.cc:407] Connection header received
0506 15:42:51.861995 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:51.862005 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:51.862021 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:51.862095 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:51.862956 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.863547 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:51.864694 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.864899 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:51.869814 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:51.869820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:51.869820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:51.869828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:51.871629 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:51.872127 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:51.872129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:51.872129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:51.872157 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:51.872355 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:51.872356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:51.872356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:51.872472 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:51.872513 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:51.872854 (+ 341us) server_negotiation.cc:299] Negotiation successful
0506 15:42:51.872967 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240506 15:42:51.873546 31516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:51.860472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:35763 (local address 127.17.240.194:43337)
0506 15:42:51.860693 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:51.860730 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:51.861842 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:51.862171 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:51.862176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:51.862442 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:51.862834 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:51.862840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.863641 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:51.863642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:51.864163 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:51.864165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:51.864750 (+ 585us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:51.868102 (+ 3352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:51.868132 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:51.869678 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:51.871741 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:51.871748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:51.871752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:51.872042 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:51.872220 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:51.872221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:51.872221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:51.872296 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:51.872554 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:51.872558 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:51.872708 (+ 150us) client_negotiation.cc:769] Sending connection context
0506 15:42:51.873222 (+ 514us) client_negotiation.cc:240] Negotiation successful
0506 15:42:51.873354 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":125,"threads_started":1}
I20240506 15:42:51.873996 31514 heartbeater.cc:347] Connected to a master server at 127.17.240.254:35763
I20240506 15:42:51.874095 31514 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:51.874311 31514 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
W20240506 15:42:51.874861 31514 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:35763 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240506 15:42:51.874940 31514 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:51.875057 31514 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
I20240506 15:42:51.875326 31514 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:51.875419 31514 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
W20240506 15:42:51.875667 31514 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240506 15:42:51.989197 31517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:51.989538 31517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:51.989624 31517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:51.992260 31517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:51.992497 31517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:51.992589 31517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:42:51.995280 31517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.240.212:39089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:40539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=42619
--tserver_master_addrs=127.17.240.254:35763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:51.995801 31517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:51.996110 31517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:52.001977 31525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:52.002588 31523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:52.003217 31517 server_base.cc:1034] running on GCE node
W20240506 15:42:52.004055 31522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:52.004317 31517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:52.004604 31517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:52.011974 31517 hybrid_clock.cc:648] HybridClock initialized: now 1715010172011954 us; error 47 us; skew 500 ppm
May 06 15:42:52 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010172, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:52.015417 31517 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:42:52.015947 31517 webserver.cc:460] Webserver started at http://127.17.240.195:42619/ using document root <none> and password file <none>
I20240506 15:42:52.016172 31517 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:52.016227 31517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 06 15:42:52 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010172, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:52.044695 31517 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:42:52.048012 31534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:52.048657 31517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20240506 15:42:52.048769 31517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "2f47e7bb6995420e90bb837433dd22a0"
format_stamp: "Formatted at 2024-05-06 15:42:51 on dist-test-slave-v0k1"
server_key: "ac41554ac27a3eb9526cca2b6408a917"
server_key_iv: "5573c208f031b23da0d170187ab139d6"
server_key_version: "kuduclusterkey@0"
I20240506 15:42:52.049194 31517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:52.067891 31517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:52.071640 31517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:52.071853 31517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:52.072160 31517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:42:52.072504 31517 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:42:52.072543 31517 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:52.072578 31517 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:42:52.072600 31517 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:52.079560 31517 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:40539
I20240506 15:42:52.079974 31517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:42:52.081405 31647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:40539 every 8 connection(s)
I20240506 15:42:52.081856 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 31517
May 06 15:42:52 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010172, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:52.092985 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:52.082115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:39961 (local address 127.17.240.254:35763)
0506 15:42:52.082388 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:52.082390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:52.083259 (+ 869us) server_negotiation.cc:407] Connection header received
0506 15:42:52.084429 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:52.084431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:52.084444 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:52.084475 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:52.085461 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:52.086228 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:52.087076 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:52.087249 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:52.089703 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:52.089707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:52.089708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:52.089716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:52.091526 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:52.092014 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:52.092016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:52.092016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:52.092043 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:52.092257 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:52.092258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:52.092258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:52.092365 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:52.092408 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:52.092781 (+ 373us) server_negotiation.cc:299] Negotiation successful
0506 15:42:52.092879 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20240506 15:42:52.093713 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:52.081084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:35763 (local address 127.17.240.195:39961)
0506 15:42:52.083139 (+ 2055us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:52.083174 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:52.084256 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:52.084673 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:52.084678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:52.084937 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:52.085345 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:52.085352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:52.086349 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:52.086350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:52.086941 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:52.086944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:52.087371 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:52.087927 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:52.087949 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:52.089590 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:52.091629 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:52.091635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:52.091639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:52.091925 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:52.092109 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:52.092109 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:52.092110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:52.092190 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:52.092452 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:52.092457 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:52.092619 (+ 162us) client_negotiation.cc:769] Sending connection context
0506 15:42:52.093355 (+ 736us) client_negotiation.cc:240] Negotiation successful
0506 15:42:52.093494 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2011,"thread_start_us":92,"threads_started":1}
I20240506 15:42:52.094374 31648 heartbeater.cc:347] Connected to a master server at 127.17.240.254:35763
I20240506 15:42:52.094535 31648 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:52.094811 31648 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
W20240506 15:42:52.095450 31648 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:35763 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240506 15:42:52.095584 31648 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:52.095718 31648 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
I20240506 15:42:52.096045 31648 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:52.096189 31648 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
W20240506 15:42:52.096508 31648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240506 15:42:52.110576 31652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:52.082292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.17.240.254:35763)
0506 15:42:52.083996 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:52.083999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:52.084007 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:42:52.084034 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:52.084035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:52.084048 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:52.090489 (+ 6441us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:52.091555 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:52.092337 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:52.106834 (+ 14497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:52.107076 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:52.107150 (+ 74us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:52.107156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:52.107156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:52.107161 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:52.109216 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:52.109736 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:52.109737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:52.109738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:52.109768 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:52.109958 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:52.109959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:52.109959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:52.110082 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:52.110241 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:52.110362 (+ 121us) server_negotiation.cc:299] Negotiation successful
0506 15:42:52.110462 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1681,"thread_start_us":65,"threads_started":1}
May 06 15:42:52 dist-test-slave-v0k1 krb5kdc[30186](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1715010172, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:52.659665 31380 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:52.659906 31380 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
I20240506 15:42:52.660336 31182 ts_manager.cc:194] Registered new tserver with Master: 0177154837644b34be29eee40aaea3ab (127.17.240.193:44383)
I20240506 15:42:52.661401 31182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:47645
I20240506 15:42:52.875828 31514 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:52.876034 31514 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
I20240506 15:42:52.876356 31182 ts_manager.cc:194] Registered new tserver with Master: fa361494b6b44c098200535727c6afd2 (127.17.240.194:46413)
I20240506 15:42:52.876765 31182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:43337
May 06 15:42:53 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010172, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
I20240506 15:42:53.096722 31648 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:53.096951 31648 heartbeater.cc:511] Master 127.17.240.254:35763 requested a full tablet report, sending...
I20240506 15:42:53.097272 31182 ts_manager.cc:194] Registered new tserver with Master: 2f47e7bb6995420e90bb837433dd22a0 (127.17.240.195:40539)
I20240506 15:42:53.097800 31182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:39961
I20240506 15:42:53.101725 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:42:53.110070 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.102740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.17.240.254:35763)
0506 15:42:53.102858 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:53.102860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:53.102911 (+ 51us) server_negotiation.cc:407] Connection header received
0506 15:42:53.102991 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:53.102992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:53.103018 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:53.103079 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:53.103979 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.104749 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.105543 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.105747 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.106865 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:53.106870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:53.106871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:53.106878 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:53.108846 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:53.109240 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:53.109241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:53.109242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:53.109270 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:53.109452 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:53.109453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:53.109454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:53.109574 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:53.109677 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:53.109830 (+ 153us) server_negotiation.cc:299] Negotiation successful
0506 15:42:53.109964 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20240506 15:42:53.112536 31182 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:42:53.114650 31182 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:42:53.121624 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.117465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:40539 (local address 127.0.0.1:41428)
0506 15:42:53.117622 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:53.117650 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:53.117729 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:53.118448 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:53.118452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:53.118454 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:42:53.118707 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:53.118709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.120417 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.120418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:53.121307 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:53.121309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.121360 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.121399 (+ 39us) client_negotiation.cc:769] Sending connection context
0506 15:42:53.121444 (+ 45us) client_negotiation.cc:240] Negotiation successful
0506 15:42:53.121490 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20240506 15:42:53.122120 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.117738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:44383 (local address 127.0.0.1:34708)
0506 15:42:53.117857 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:53.117865 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:53.117906 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:53.118747 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:53.118748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:53.118750 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:42:53.118958 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:53.118960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.120364 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.120366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:53.121302 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:53.121305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.121406 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.121411 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:42:53.121445 (+ 34us) client_negotiation.cc:240] Negotiation successful
0506 15:42:53.121497 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":78,"threads_started":1}
I20240506 15:42:53.122625 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.117545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.17.240.195:40539)
0506 15:42:53.118246 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:53.118251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:53.118263 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:42:53.118321 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:53.118323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:53.118340 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:53.118440 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:42:53.118795 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.120329 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.121623 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.122119 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.122169 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:53.122203 (+ 34us) server_negotiation.cc:299] Negotiation successful
0506 15:42:53.122236 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":61,"threads_started":1}
I20240506 15:42:53.123236 31564 tablet_service.cc:1467] Processing CreateTablet for tablet af6c2decd51d4cb1bcde228a7f2fce1f (DEFAULT_TABLE table=test-table [id=30c55c31fd3e4c4b85bc34bb27ac1f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:42:53.123373 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.118095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.17.240.193:44383)
0506 15:42:53.118575 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:53.118579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:53.118588 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:42:53.118623 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:53.118625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:53.118641 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:53.119074 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:42:53.119148 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.120243 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.122456 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.123172 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.123216 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:53.123249 (+ 33us) server_negotiation.cc:299] Negotiation successful
0506 15:42:53.123282 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":61,"threads_started":1}
I20240506 15:42:53.123554 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af6c2decd51d4cb1bcde228a7f2fce1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:53.124126 31314 tablet_service.cc:1467] Processing CreateTablet for tablet af6c2decd51d4cb1bcde228a7f2fce1f (DEFAULT_TABLE table=test-table [id=30c55c31fd3e4c4b85bc34bb27ac1f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:42:53.124415 31314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af6c2decd51d4cb1bcde228a7f2fce1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:53.127164 31681 tablet_bootstrap.cc:492] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: Bootstrap starting.
I20240506 15:42:53.127735 31681 tablet_bootstrap.cc:654] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:53.127879 31682 tablet_bootstrap.cc:492] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab: Bootstrap starting.
I20240506 15:42:53.128048 31681 log.cc:824] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:53.128609 31682 tablet_bootstrap.cc:654] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:53.128939 31682 log.cc:824] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:53.129791 31682 tablet_bootstrap.cc:492] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab: No bootstrap required, opened a new log
I20240506 15:42:53.129902 31682 ts_tablet_manager.cc:1367] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:42:53.131031 31681 tablet_bootstrap.cc:492] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: No bootstrap required, opened a new log
I20240506 15:42:53.131106 31681 ts_tablet_manager.cc:1367] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240506 15:42:53.132012 31682 raft_consensus.cc:348] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 } } peers { permanent_uuid: "0177154837644b34be29eee40aaea3ab" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44383 } } peers { permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 } }
I20240506 15:42:53.132181 31682 raft_consensus.cc:374] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:53.132205 31682 raft_consensus.cc:729] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0177154837644b34be29eee40aaea3ab, State: Initialized, Role: FOLLOWER
I20240506 15:42:53.132318 31682 consensus_queue.cc:260] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 } } peers { permanent_uuid: "0177154837644b34be29eee40aaea3ab" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44383 } } peers { permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 } }
I20240506 15:42:53.132643 31682 ts_tablet_manager.cc:1398] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 15:42:53.132686 31380 heartbeater.cc:503] Master 127.17.240.254:35763 was elected leader, sending a full tablet report...
I20240506 15:42:53.132912 31681 raft_consensus.cc:348] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 } } peers { permanent_uuid: "0177154837644b34be29eee40aaea3ab" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44383 } } peers { permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 } }
I20240506 15:42:53.133044 31681 raft_consensus.cc:374] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:53.133074 31681 raft_consensus.cc:729] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f47e7bb6995420e90bb837433dd22a0, State: Initialized, Role: FOLLOWER
I20240506 15:42:53.133167 31681 consensus_queue.cc:260] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 } } peers { permanent_uuid: "0177154837644b34be29eee40aaea3ab" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44383 } } peers { permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 } }
I20240506 15:42:53.133404 31681 ts_tablet_manager.cc:1398] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:42:53.133615 31648 heartbeater.cc:503] Master 127.17.240.254:35763 was elected leader, sending a full tablet report...
I20240506 15:42:53.136175 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.117908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:46413 (local address 127.0.0.1:41506)
0506 15:42:53.118003 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:53.118011 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:53.118311 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:53.125782 (+ 7471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:53.125784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:53.125787 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:42:53.125996 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:53.125999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.135164 (+ 9165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.135165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:53.135986 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:53.135990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.136055 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.136060 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:42:53.136077 (+ 17us) client_negotiation.cc:240] Negotiation successful
0506 15:42:53.136115 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":67,"threads_started":1}
I20240506 15:42:53.136862 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.118004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.17.240.194:46413)
0506 15:42:53.125515 (+ 7511us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:53.125520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:53.125532 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:42:53.125575 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:53.125577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:53.125597 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:53.125692 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:42:53.126098 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.127234 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.136133 (+ 8899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.136646 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.136698 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:53.136735 (+ 37us) server_negotiation.cc:299] Negotiation successful
0506 15:42:53.136771 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7461,"thread_start_us":52,"threads_started":1}
I20240506 15:42:53.137511 31448 tablet_service.cc:1467] Processing CreateTablet for tablet af6c2decd51d4cb1bcde228a7f2fce1f (DEFAULT_TABLE table=test-table [id=30c55c31fd3e4c4b85bc34bb27ac1f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:42:53.137825 31448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af6c2decd51d4cb1bcde228a7f2fce1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 15:42:53.139933 31381 tablet.cc:2369] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:42:53.141058 31687 tablet_bootstrap.cc:492] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: Bootstrap starting.
I20240506 15:42:53.141610 31687 tablet_bootstrap.cc:654] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:53.141834 31687 log.cc:824] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:53.142482 31687 tablet_bootstrap.cc:492] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: No bootstrap required, opened a new log
I20240506 15:42:53.142550 31687 ts_tablet_manager.cc:1367] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:42:53.144049 31687 raft_consensus.cc:348] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 } } peers { permanent_uuid: "0177154837644b34be29eee40aaea3ab" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44383 } } peers { permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 } }
I20240506 15:42:53.144179 31687 raft_consensus.cc:374] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:53.144197 31687 raft_consensus.cc:729] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa361494b6b44c098200535727c6afd2, State: Initialized, Role: FOLLOWER
I20240506 15:42:53.144287 31687 consensus_queue.cc:260] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 } } peers { permanent_uuid: "0177154837644b34be29eee40aaea3ab" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44383 } } peers { permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 } }
I20240506 15:42:53.144536 31687 ts_tablet_manager.cc:1398] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:42:53.144600 31514 heartbeater.cc:503] Master 127.17.240.254:35763 was elected leader, sending a full tablet report...
May 06 15:42:53 dist-test-slave-v0k1 krb5kdc[30186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010172, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for HTTP/127.17.240.212@KRBTEST.COM
W20240506 15:42:53.331430 31649 tablet.cc:2369] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:42:53.360592 31685 raft_consensus.cc:482] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:42:53.360749 31685 raft_consensus.cc:504] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 } } peers { permanent_uuid: "0177154837644b34be29eee40aaea3ab" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44383 } } peers { permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 } }
I20240506 15:42:53.361112 31685 leader_election.cc:288] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f47e7bb6995420e90bb837433dd22a0 (127.17.240.195:40539), fa361494b6b44c098200535727c6afd2 (127.17.240.194:46413)
W20240506 15:42:53.361119 31515 tablet.cc:2369] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:42:53.365504 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.361828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:39345 (local address 127.17.240.194:46413)
0506 15:42:53.361870 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:53.361872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:53.362149 (+ 277us) server_negotiation.cc:407] Connection header received
0506 15:42:53.362256 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:53.362257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:53.362267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:53.362300 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:42:53.362879 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.363638 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.364783 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.365240 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.365249 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:53.365365 (+ 116us) server_negotiation.cc:299] Negotiation successful
0506 15:42:53.365428 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240506 15:42:53.365751 31692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.361566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:46413 (local address 127.17.240.193:39345)
0506 15:42:53.362092 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:53.362100 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:53.362174 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:53.362386 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:53.362387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:53.362389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:42:53.362731 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:53.362734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.363741 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.363742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:53.364697 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:53.364698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.365276 (+ 578us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.365280 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:42:53.365632 (+ 352us) client_negotiation.cc:240] Negotiation successful
0506 15:42:53.365673 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":64,"threads_started":1}
I20240506 15:42:53.366045 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6c2decd51d4cb1bcde228a7f2fce1f" candidate_uuid: "0177154837644b34be29eee40aaea3ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa361494b6b44c098200535727c6afd2" is_pre_election: true
I20240506 15:42:53.366209 31464 raft_consensus.cc:2454] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0177154837644b34be29eee40aaea3ab in term 0.
I20240506 15:42:53.366433 31267 leader_election.cc:302] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0177154837644b34be29eee40aaea3ab, fa361494b6b44c098200535727c6afd2; no voters:
I20240506 15:42:53.366570 31685 raft_consensus.cc:2781] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:42:53.366626 31685 raft_consensus.cc:482] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:42:53.366647 31685 raft_consensus.cc:3037] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:53.367240 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.361767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:59855 (local address 127.17.240.195:40539)
0506 15:42:53.361820 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:53.361822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:53.363144 (+ 1322us) server_negotiation.cc:407] Connection header received
0506 15:42:53.363174 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:53.363175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:53.363185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:53.363242 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:42:53.363633 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.364362 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.366416 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.366958 (+ 542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.366967 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:53.367123 (+ 156us) server_negotiation.cc:299] Negotiation successful
0506 15:42:53.367171 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240506 15:42:53.367609 31685 raft_consensus.cc:504] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 } } peers { permanent_uuid: "0177154837644b34be29eee40aaea3ab" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44383 } } peers { permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 } }
I20240506 15:42:53.367746 31685 leader_election.cc:288] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [CANDIDATE]: Term 1 election: Requested vote from peers 2f47e7bb6995420e90bb837433dd22a0 (127.17.240.195:40539), fa361494b6b44c098200535727c6afd2 (127.17.240.194:46413)
I20240506 15:42:53.368065 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6c2decd51d4cb1bcde228a7f2fce1f" candidate_uuid: "0177154837644b34be29eee40aaea3ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa361494b6b44c098200535727c6afd2"
I20240506 15:42:53.368139 31464 raft_consensus.cc:3037] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:53.369017 31464 raft_consensus.cc:2454] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0177154837644b34be29eee40aaea3ab in term 1.
I20240506 15:42:53.369225 31267 leader_election.cc:302] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0177154837644b34be29eee40aaea3ab, fa361494b6b44c098200535727c6afd2; no voters:
I20240506 15:42:53.369537 31685 raft_consensus.cc:2781] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:42:53.369694 31685 raft_consensus.cc:686] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [term 1 LEADER]: Becoming Leader. State: Replica: 0177154837644b34be29eee40aaea3ab, State: Running, Role: LEADER
I20240506 15:42:53.369787 31685 consensus_queue.cc:237] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 } } peers { permanent_uuid: "0177154837644b34be29eee40aaea3ab" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44383 } } peers { permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 } }
I20240506 15:42:53.370632 31182 catalog_manager.cc:5484] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 0177154837644b34be29eee40aaea3ab (127.17.240.193). New cstate: current_term: 1 leader_uuid: "0177154837644b34be29eee40aaea3ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0177154837644b34be29eee40aaea3ab" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 44383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 } health_report { overall_health: UNKNOWN } } }
I20240506 15:42:53.372231 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.361347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:40539 (local address 127.17.240.193:59855)
0506 15:42:53.361931 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:53.361943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:53.362010 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:53.363293 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:53.363294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:53.363296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:42:53.363544 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:53.363546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.364473 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.364474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:53.365344 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:53.365346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.366998 (+ 1652us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.367002 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:42:53.372082 (+ 5080us) client_negotiation.cc:240] Negotiation successful
0506 15:42:53.372150 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":93,"threads_started":1}
I20240506 15:42:53.372642 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6c2decd51d4cb1bcde228a7f2fce1f" candidate_uuid: "0177154837644b34be29eee40aaea3ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f47e7bb6995420e90bb837433dd22a0" is_pre_election: true
I20240506 15:42:53.372644 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6c2decd51d4cb1bcde228a7f2fce1f" candidate_uuid: "0177154837644b34be29eee40aaea3ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f47e7bb6995420e90bb837433dd22a0"
I20240506 15:42:53.372822 31602 raft_consensus.cc:2454] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0177154837644b34be29eee40aaea3ab in term 0.
I20240506 15:42:53.418659 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.415639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.17.240.193:44383)
0506 15:42:53.415684 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:53.415686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:53.415694 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:42:53.415750 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:53.415751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:53.415773 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:53.415851 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:42:53.416256 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.416964 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:53.417859 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:53.418044 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:53.418100 (+ 56us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:42:53.418229 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:42:53.418308 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:53.418548 (+ 240us) server_negotiation.cc:299] Negotiation successful
0506 15:42:53.418578 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:42:53.427687 31464 raft_consensus.cc:1261] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2 [term 1 FOLLOWER]: Refusing update from remote peer 0177154837644b34be29eee40aaea3ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:42:53.428009 31601 raft_consensus.cc:3037] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:53.429136 31601 raft_consensus.cc:1261] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0 [term 1 FOLLOWER]: Refusing update from remote peer 0177154837644b34be29eee40aaea3ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:42:53.429487 31694 consensus_queue.cc:1035] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f47e7bb6995420e90bb837433dd22a0" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 40539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:42:53.429910 31694 consensus_queue.cc:1035] T af6c2decd51d4cb1bcde228a7f2fce1f P 0177154837644b34be29eee40aaea3ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa361494b6b44c098200535727c6afd2" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 46413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:42:53.437399 31697 mvcc.cc:205] Tried to move back new op lower bound from 7024681670353367040 to 7024681670123139072. Current Snapshot: MvccSnapshot[applied={T|T < 7024681670353367040}]
I20240506 15:42:53.439366 31698 mvcc.cc:205] Tried to move back new op lower bound from 7024681670353367040 to 7024681670123139072. Current Snapshot: MvccSnapshot[applied={T|T < 7024681670353367040}]
I20240506 15:42:53.445773 31700 mvcc.cc:205] Tried to move back new op lower bound from 7024681670353367040 to 7024681670123139072. Current Snapshot: MvccSnapshot[applied={T|T < 7024681670353367040}]
I20240506 15:42:53.453930 31182 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44208:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:42:53.454061 31182 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44208:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:42:53.454943 31182 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 35a681f18ea244ccab3258c090f60d21: Sending DeleteTablet for 3 replicas of tablet af6c2decd51d4cb1bcde228a7f2fce1f
I20240506 15:42:53.455447 31314 tablet_service.cc:1514] Processing DeleteTablet for tablet af6c2decd51d4cb1bcde228a7f2fce1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:42:53 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:34708
I20240506 15:42:53.455610 31564 tablet_service.cc:1514] Processing DeleteTablet for tablet af6c2decd51d4cb1bcde228a7f2fce1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:42:53 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:41428
I20240506 15:42:53.455714 31448 tablet_service.cc:1514] Processing DeleteTablet for tablet af6c2decd51d4cb1bcde228a7f2fce1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:42:53 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:41506
I20240506 15:42:53.456089 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 31220
I20240506 15:42:53.463155 31707 tablet_replica.cc:330] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: stopping tablet replica
I20240506 15:42:53.463300 31707 raft_consensus.cc:2229] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:42:53.463450 31707 raft_consensus.cc:2258] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:42:53.463524 31708 tablet_replica.cc:330] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: stopping tablet replica
I20240506 15:42:53.463711 31708 raft_consensus.cc:2229] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:42:53.463809 31708 raft_consensus.cc:2258] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:42:53.464334 31708 ts_tablet_manager.cc:1875] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:42:53.464577 31707 ts_tablet_manager.cc:1875] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:42:53.466621 31708 ts_tablet_manager.cc:1888] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:42:53.466780 31708 log.cc:1196] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/af6c2decd51d4cb1bcde228a7f2fce1f
I20240506 15:42:53.467200 31708 ts_tablet_manager.cc:1909] T af6c2decd51d4cb1bcde228a7f2fce1f P fa361494b6b44c098200535727c6afd2: Deleting consensus metadata
I20240506 15:42:53.467743 31166 catalog_manager.cc:4830] TS fa361494b6b44c098200535727c6afd2 (127.17.240.194:46413): tablet af6c2decd51d4cb1bcde228a7f2fce1f (table test-table [id=30c55c31fd3e4c4b85bc34bb27ac1f64]) successfully deleted
I20240506 15:42:53.471230 31707 ts_tablet_manager.cc:1888] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:42:53.471338 31707 log.cc:1196] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/af6c2decd51d4cb1bcde228a7f2fce1f
I20240506 15:42:53.471673 31707 ts_tablet_manager.cc:1909] T af6c2decd51d4cb1bcde228a7f2fce1f P 2f47e7bb6995420e90bb837433dd22a0: Deleting consensus metadata
I20240506 15:42:53.472157 31168 catalog_manager.cc:4830] TS 2f47e7bb6995420e90bb837433dd22a0 (127.17.240.195:40539): tablet af6c2decd51d4cb1bcde228a7f2fce1f (table test-table [id=30c55c31fd3e4c4b85bc34bb27ac1f64]) successfully deleted
W20240506 15:42:53.479346 31166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:44383 (error 108)
I20240506 15:42:53.485488 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 31384
I20240506 15:42:53.486456 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:53.486314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:44383 (local address 127.0.0.1:34736)
0506 15:42:53.486360 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:53.486412 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20240506 15:42:53.486586 31166 catalog_manager.cc:4573] TS 0177154837644b34be29eee40aaea3ab (127.17.240.193:44383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af6c2decd51d4cb1bcde228a7f2fce1f: Network error: Client connection negotiation failed: client connection to 127.17.240.193:44383: connect: Connection refused (error 111)
I20240506 15:42:53.490571 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 31517
I20240506 15:42:53.498713 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 31149
I20240506 15:42:53.505352 18371 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240506 15:42:53.850327 18371 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240506 15:42:53.850427 18371 mini_ranger.cc:66] Stopping Ranger...
I20240506 15:42:54.195645 18371 mini_ranger.cc:68] Stopped Ranger
2024-05-06 15:42:54.195 UTC [30216] LOG: received smart shutdown request
2024-05-06 15:42:54.199 UTC [30216] LOG: background worker "logical replication launcher" (PID 30225) exited with exit code 1
2024-05-06 15:42:54.200 UTC [30220] LOG: shutting down
2024-05-06 15:42:54.383 UTC [30216] LOG: database system is shut down
2024-05-06T15:42:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48622 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:42:54 dist-test-slave-v0k1 krb5kdc[31718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:42:54 dist-test-slave-v0k1 krb5kdc[31718](info): set up 2 sockets
May 06 15:42:54 dist-test-slave-v0k1 krb5kdc[31718](info): commencing operation
krb5kdc: starting...
W20240506 15:42:55.447057 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:42:55 dist-test-slave-v0k1 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:42:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:42:55.566673 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39119
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:39119,127.17.240.253:38205,127.17.240.252:35319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:55.646226 31731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:55.646420 31731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:55.646443 31731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:55.647728 31731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:42:55.647768 31731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:55.647781 31731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:55.647794 31731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:42:55.647804 31731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:42:55.649333 31731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:39119,127.17.240.253:38205,127.17.240.252:35319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:55.649518 31731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:55.649662 31731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:55.654384 31739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:55.654752 31737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:55.655635 31736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:55.656473 31731 server_base.cc:1034] running on GCE node
I20240506 15:42:55.656746 31731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:55.656992 31731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:55.658140 31731 hybrid_clock.cc:648] HybridClock initialized: now 1715010175658121 us; error 33 us; skew 500 ppm
May 06 15:42:55 dist-test-slave-v0k1 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:55.665236 31731 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:42:55.665657 31731 webserver.cc:460] Webserver started at http://127.17.240.254:39681/ using document root <none> and password file <none>
I20240506 15:42:55.665843 31731 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:55.665886 31731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:55.666014 31731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:42:55.667052 31731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "b5479e35fecb4829aac126b22cce94ab"
format_stamp: "Formatted at 2024-05-06 15:42:55 on dist-test-slave-v0k1"
server_key: "5c75232ed67eb047fe7dbd56932051b5"
server_key_iv: "654218ec0bef5685af3bc8366e73c8fa"
server_key_version: "encryptionkey@0"
I20240506 15:42:55.667358 31731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "b5479e35fecb4829aac126b22cce94ab"
format_stamp: "Formatted at 2024-05-06 15:42:55 on dist-test-slave-v0k1"
server_key: "5c75232ed67eb047fe7dbd56932051b5"
server_key_iv: "654218ec0bef5685af3bc8366e73c8fa"
server_key_version: "encryptionkey@0"
I20240506 15:42:55.668540 31731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:42:55.669291 31746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:55.669500 31731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:42:55.669577 31731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "b5479e35fecb4829aac126b22cce94ab"
format_stamp: "Formatted at 2024-05-06 15:42:55 on dist-test-slave-v0k1"
server_key: "5c75232ed67eb047fe7dbd56932051b5"
server_key_iv: "654218ec0bef5685af3bc8366e73c8fa"
server_key_version: "encryptionkey@0"
I20240506 15:42:55.669636 31731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:55.676342 31731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:55.678637 31731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:55.678766 31731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:55.682729 31731 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:39119
I20240506 15:42:55.682780 31798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:39119 every 8 connection(s)
I20240506 15:42:55.683109 31731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:42:55.683671 31799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:55.685494 31799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:42:55 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:55.691536 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 31731
I20240506 15:42:55.691648 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:42:55.691856 18371 external_mini_cluster.cc:1428] Setting key 765f0904fc549a6dd457977cb90a7b9f
I20240506 15:42:55.694449 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.686173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.17.240.254:39119)
0506 15:42:55.686336 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:55.686348 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:55.686483 (+ 135us) server_negotiation.cc:407] Connection header received
0506 15:42:55.686991 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:55.686993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:55.687040 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:55.687123 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:55.687812 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.688514 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.689017 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.689198 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:55.691172 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:55.691183 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:55.691184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:55.691189 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:55.692965 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:55.693409 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:55.693412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:55.693413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:55.693459 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:55.693693 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:55.693694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:55.693695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:55.693882 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:55.693961 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:55.694168 (+ 207us) server_negotiation.cc:299] Negotiation successful
0506 15:42:55.694272 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20240506 15:42:55.694561 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.686126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39119 (local address 127.0.0.1:33360)
0506 15:42:55.686331 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.686361 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:55.686849 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:55.687140 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:55.687142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:55.687477 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:55.687716 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:55.687720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.688594 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.688595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:55.688945 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:55.688946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.689085 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:55.689673 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:55.689689 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:55.691094 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:55.693046 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:55.693051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:55.693055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:55.693294 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:55.693531 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:55.693531 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:55.693532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:55.693597 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:55.693957 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:55.693962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:55.694040 (+ 78us) client_negotiation.cc:769] Sending connection context
0506 15:42:55.694147 (+ 107us) client_negotiation.cc:240] Negotiation successful
0506 15:42:55.694380 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":78,"thread_start_us":100,"threads_started":1}
I20240506 15:42:55.695385 31799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:42:55.695740 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.695602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38205 (local address 127.0.0.1:52126)
0506 15:42:55.695655 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.695705 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240506 15:42:55.695914 31748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.240.253:38205: connect: Connection refused (error 111)
I20240506 15:42:55.696409 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.696267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38205 (local address 127.0.0.1:52128)
0506 15:42:55.696314 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.696374 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240506 15:42:55.696522 31799 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.253:38205: Network error: Client connection negotiation failed: client connection to 127.17.240.253:38205: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.17.240.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab.
Entry for principal kudu/127.17.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab.
I20240506 15:42:55.717985 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.253:38205
--webserver_interface=127.17.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab
--principal=kudu/127.17.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:39119,127.17.240.253:38205,127.17.240.252:35319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:42:55.745745 31799 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } attempt: 1
I20240506 15:42:55.746309 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.746171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38205 (local address 127.0.0.1:52142)
0506 15:42:55.746239 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.746273 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20240506 15:42:55.746711 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.746635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38205 (local address 127.0.0.1:52150)
0506 15:42:55.746671 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.746692 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240506 15:42:55.746799 31799 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.253:38205: Network error: Client connection negotiation failed: client connection to 127.17.240.253:38205: connect: Connection refused (error 111)
W20240506 15:42:55.798803 31806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:55.798995 31806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:55.799038 31806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:55.800343 31806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:42:55.800396 31806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:55.800411 31806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:55.800423 31806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:42:55.800434 31806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:42:55.802026 31806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:39119,127.17.240.253:38205,127.17.240.252:35319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab
--principal=kudu/127.17.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.253:38205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:55.802250 31806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:55.802454 31806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:55.806838 31811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:55.807641 31812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:55.808036 31814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:55.808651 31806 server_base.cc:1034] running on GCE node
I20240506 15:42:55.808833 31806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:55.809053 31806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:55.810190 31806 hybrid_clock.cc:648] HybridClock initialized: now 1715010175810171 us; error 38 us; skew 500 ppm
May 06 15:42:55 dist-test-slave-v0k1 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:55.812592 31806 init.cc:377] Logged in from keytab as kudu/127.17.240.253@KRBTEST.COM (short username kudu)
I20240506 15:42:55.812994 31806 webserver.cc:460] Webserver started at http://127.17.240.253:34597/ using document root <none> and password file <none>
I20240506 15:42:55.813196 31806 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:55.813239 31806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:55.813329 31806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:42:55.814355 31806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/data/instance:
uuid: "1206a7560fe44da5ae596d3164aba355"
format_stamp: "Formatted at 2024-05-06 15:42:55 on dist-test-slave-v0k1"
server_key: "4c6b1e85f2a64a343353d9ebf6212205"
server_key_iv: "56d43ad578b5b9f10f7ae996f839828e"
server_key_version: "encryptionkey@0"
I20240506 15:42:55.814662 31806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/wal/instance:
uuid: "1206a7560fe44da5ae596d3164aba355"
format_stamp: "Formatted at 2024-05-06 15:42:55 on dist-test-slave-v0k1"
server_key: "4c6b1e85f2a64a343353d9ebf6212205"
server_key_iv: "56d43ad578b5b9f10f7ae996f839828e"
server_key_version: "encryptionkey@0"
I20240506 15:42:55.814955 31799 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } attempt: 2
I20240506 15:42:55.815387 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.815233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38205 (local address 127.0.0.1:52162)
0506 15:42:55.815302 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.815343 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20240506 15:42:55.815834 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.815714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38205 (local address 127.0.0.1:52170)
0506 15:42:55.815772 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.815802 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20240506 15:42:55.815945 31799 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.253:38205: Network error: Client connection negotiation failed: client connection to 127.17.240.253:38205: connect: Connection refused (error 111)
I20240506 15:42:55.816184 31806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:42:55.816967 31821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:55.817147 31806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:42:55.817219 31806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/wal
uuid: "1206a7560fe44da5ae596d3164aba355"
format_stamp: "Formatted at 2024-05-06 15:42:55 on dist-test-slave-v0k1"
server_key: "4c6b1e85f2a64a343353d9ebf6212205"
server_key_iv: "56d43ad578b5b9f10f7ae996f839828e"
server_key_version: "encryptionkey@0"
I20240506 15:42:55.817294 31806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:55.826789 31806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:55.829077 31806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:55.829192 31806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:55.833199 31806 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.253:38205
I20240506 15:42:55.833233 31873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.253:38205 every 8 connection(s)
I20240506 15:42:55.833534 31806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
I20240506 15:42:55.834113 31874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:55.835984 31874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:42:55 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:55.842949 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 31806
I20240506 15:42:55.843111 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-1/wal/instance
I20240506 15:42:55.843317 18371 external_mini_cluster.cc:1428] Setting key 664134afd88c601e1979f3c1dc0b082f
I20240506 15:42:55.844030 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.836673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.17.240.254:39119)
0506 15:42:55.836739 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:55.836742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:55.836888 (+ 146us) server_negotiation.cc:407] Connection header received
0506 15:42:55.837330 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:55.837331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:55.837341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:55.837443 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:55.838166 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.838624 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.839187 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.839305 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:55.841289 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:55.841294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:55.841295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:55.841301 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:55.842607 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:55.843102 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:55.843104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:55.843104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:55.843134 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:55.843374 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:55.843374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:55.843375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:55.843498 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:55.843554 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:55.843883 (+ 329us) server_negotiation.cc:299] Negotiation successful
0506 15:42:55.843962 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:42:55.844276 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.836628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39119 (local address 127.0.0.1:33368)
0506 15:42:55.836781 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.836813 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:55.837247 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:55.837491 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:55.837502 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:55.837796 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:55.838087 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:55.838091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.838701 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.838702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:55.839127 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:55.839129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.839242 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:55.839798 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:55.839815 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:55.841190 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:55.842685 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:55.842690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:55.842693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:55.842996 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:55.843209 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:55.843209 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:55.843210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:55.843309 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:55.843568 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:55.843575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:55.843738 (+ 163us) client_negotiation.cc:769] Sending connection context
0506 15:42:55.843872 (+ 134us) client_negotiation.cc:240] Negotiation successful
0506 15:42:55.844031 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20240506 15:42:55.845376 31874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:42:55 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.240.252@KRBTEST.COM; defaulting to no policy
I20240506 15:42:55.852916 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.845657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38205 (local address 127.0.0.1:52172)
0506 15:42:55.845713 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.845723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:55.845755 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:55.846017 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:55.846018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:55.846192 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:55.846402 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:55.846403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.847340 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.847342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:55.847950 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:55.847952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.848040 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:55.848559 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:55.848564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:55.849963 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:55.852076 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:55.852077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:55.852078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:55.852215 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:55.852360 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:55.852361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:55.852361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:55.852381 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:55.852590 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:55.852591 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:55.852627 (+ 36us) client_negotiation.cc:769] Sending connection context
0506 15:42:55.852693 (+ 66us) client_negotiation.cc:240] Negotiation successful
0506 15:42:55.852774 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20240506 15:42:55.853001 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.845721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.17.240.253:38205)
0506 15:42:55.845883 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:55.845891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:55.845906 (+ 15us) server_negotiation.cc:407] Connection header received
0506 15:42:55.845925 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:55.845926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:55.845937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:55.845997 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:55.846489 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.847267 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.848052 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.848240 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:55.850102 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:55.850113 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:55.850114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:55.850120 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:55.851968 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:55.852278 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:55.852280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:55.852280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:55.852309 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:55.852427 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:55.852427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:55.852427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:55.852535 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:55.852613 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:55.852725 (+ 112us) server_negotiation.cc:299] Negotiation successful
0506 15:42:55.852830 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.252@KRBTEST.COM" created.
I20240506 15:42:55.853582 31874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:42:55.853937 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.853802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41658)
0506 15:42:55.853851 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.853905 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240506 15:42:55.855597 31824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
I20240506 15:42:55.856051 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.855953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41666)
0506 15:42:55.855997 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.856024 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240506 15:42:55.856173 31874 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:35319: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab.
Entry for principal kudu/127.17.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab.
I20240506 15:42:55.869930 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.252:35319
--webserver_interface=127.17.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab
--principal=kudu/127.17.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:39119,127.17.240.253:38205,127.17.240.252:35319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:42:55.905429 31874 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } attempt: 1
I20240506 15:42:55.905989 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.905841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41672)
0506 15:42:55.905916 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.905950 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20240506 15:42:55.906477 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.906339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41686)
0506 15:42:55.906417 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.906449 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20240506 15:42:55.906607 31874 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:35319: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
I20240506 15:42:55.907059 31799 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } attempt: 3
May 06 15:42:55 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
I20240506 15:42:55.914006 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.907482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.17.240.253:38205)
0506 15:42:55.907534 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:55.907537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:55.907549 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:42:55.907573 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:55.907574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:55.907585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:55.907640 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:55.908178 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.908655 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.909410 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.909551 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:55.911351 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:55.911356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:55.911357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:55.911364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:55.912920 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:55.913230 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:55.913231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:55.913232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:55.913254 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:55.913459 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:55.913460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:55.913461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:55.913576 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:55.913652 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:55.913856 (+ 204us) server_negotiation.cc:299] Negotiation successful
0506 15:42:55.913940 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:42:55.914009 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.907328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38205 (local address 127.0.0.1:52180)
0506 15:42:55.907396 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.907411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:55.907480 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:55.907659 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:55.907660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:55.907863 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:55.908096 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:55.908098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.908732 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.908733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:55.909307 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:55.909309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.909387 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:55.909908 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:55.909913 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:55.911250 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:55.913014 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:55.913015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:55.913016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:55.913163 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:55.913307 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:55.913308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:55.913308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:55.913329 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:55.913653 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:55.913654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:55.913713 (+ 59us) client_negotiation.cc:769] Sending connection context
0506 15:42:55.913790 (+ 77us) client_negotiation.cc:240] Negotiation successful
0506 15:42:55.913907 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20240506 15:42:55.914515 31799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:42:55.914884 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.914767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41688)
0506 15:42:55.914814 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.914856 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20240506 15:42:55.915321 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.915229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41690)
0506 15:42:55.915274 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.915299 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240506 15:42:55.915460 31799 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:35319: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
I20240506 15:42:55.946655 31799 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } attempt: 1
I20240506 15:42:55.947278 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.947100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41706)
0506 15:42:55.947181 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.947232 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20240506 15:42:55.947753 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.947652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41720)
0506 15:42:55.947698 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.947726 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240506 15:42:55.947897 31799 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:35319: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
W20240506 15:42:55.956070 31881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:55.956274 31881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:55.956298 31881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:55.957662 31881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:42:55.957713 31881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:55.957726 31881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:55.957739 31881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:42:55.957751 31881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:42:55.959528 31881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:39119,127.17.240.253:38205,127.17.240.252:35319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab
--principal=kudu/127.17.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.252:35319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:55.959754 31881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:55.959951 31881 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:55.964674 31889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:55.965502 31887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:55.966209 31886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:55.966497 31881 server_base.cc:1034] running on GCE node
I20240506 15:42:55.966713 31881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:55.966961 31881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:55.968109 31881 hybrid_clock.cc:648] HybridClock initialized: now 1715010175968092 us; error 37 us; skew 500 ppm
May 06 15:42:55 dist-test-slave-v0k1 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:55.970574 31881 init.cc:377] Logged in from keytab as kudu/127.17.240.252@KRBTEST.COM (short username kudu)
I20240506 15:42:55.970984 31881 webserver.cc:460] Webserver started at http://127.17.240.252:45091/ using document root <none> and password file <none>
I20240506 15:42:55.971194 31881 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:55.971232 31881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:55.971331 31881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:42:55.972348 31881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/data/instance:
uuid: "9503191478014f839deb279978f4ce67"
format_stamp: "Formatted at 2024-05-06 15:42:55 on dist-test-slave-v0k1"
server_key: "42ca10e8fe64e3608558f2f4646fb1b1"
server_key_iv: "558e35aebb26b2bbe34eec516419a604"
server_key_version: "encryptionkey@0"
I20240506 15:42:55.972648 31881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/wal/instance:
uuid: "9503191478014f839deb279978f4ce67"
format_stamp: "Formatted at 2024-05-06 15:42:55 on dist-test-slave-v0k1"
server_key: "42ca10e8fe64e3608558f2f4646fb1b1"
server_key_iv: "558e35aebb26b2bbe34eec516419a604"
server_key_version: "encryptionkey@0"
I20240506 15:42:55.973842 31881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:42:55.974542 31896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:55.974711 31881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:42:55.974790 31881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/wal
uuid: "9503191478014f839deb279978f4ce67"
format_stamp: "Formatted at 2024-05-06 15:42:55 on dist-test-slave-v0k1"
server_key: "42ca10e8fe64e3608558f2f4646fb1b1"
server_key_iv: "558e35aebb26b2bbe34eec516419a604"
server_key_version: "encryptionkey@0"
I20240506 15:42:55.974797 31874 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } attempt: 2
I20240506 15:42:55.974879 31881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:55.975225 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.975085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41730)
0506 15:42:55.975145 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.975186 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20240506 15:42:55.975603 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.975513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41736)
0506 15:42:55.975553 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.975580 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20240506 15:42:55.975714 31874 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:35319: Network error: Client connection negotiation failed: client connection to 127.17.240.252:35319: connect: Connection refused (error 111)
I20240506 15:42:55.987169 31881 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:55.989377 31881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:55.989487 31881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:55.993436 31881 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.252:35319
I20240506 15:42:55.993476 31948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.252:35319 every 8 connection(s)
I20240506 15:42:55.993793 31881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
I20240506 15:42:55.994397 31949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:55.994977 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 31881
I20240506 15:42:55.995110 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715009978772892-18371-0/minicluster-data/master-2/wal/instance
I20240506 15:42:55.995366 18371 external_mini_cluster.cc:1428] Setting key 68e03ac2d44ec94aaf72d8de4e459b9b
I20240506 15:42:55.996547 31949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:42:55 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
May 06 15:42:55 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
I20240506 15:42:56.004848 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.997200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.17.240.254:39119)
0506 15:42:55.997262 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:55.997265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:55.997413 (+ 148us) server_negotiation.cc:407] Connection header received
0506 15:42:55.997511 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:55.997512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:55.997526 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:55.997577 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:55.998236 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.998678 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.999208 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.999381 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.001690 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.001696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.001696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.001705 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.003377 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.003953 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.003955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.003955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.003981 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.004197 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.004198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.004198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.004327 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.004391 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.004636 (+ 245us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.004737 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:42:56.005012 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.997116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39119 (local address 127.0.0.1:33374)
0506 15:42:55.997287 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:55.997314 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:55.997433 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:55.997604 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:55.997606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:55.997909 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:55.998128 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:55.998131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.998754 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.998755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:55.999122 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:55.999124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.999259 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.000074 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:56.000086 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:56.001567 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:56.003470 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:56.003476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:56.003479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:56.003878 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:56.004042 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:56.004042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:56.004043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:56.004129 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:56.004393 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:56.004398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:56.004512 (+ 114us) client_negotiation.cc:769] Sending connection context
0506 15:42:56.004606 (+ 94us) client_negotiation.cc:240] Negotiation successful
0506 15:42:56.004759 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":109,"threads_started":1}
I20240506 15:42:56.005059 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:55.995917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.17.240.252:35319)
0506 15:42:55.996147 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:55.996156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:55.996175 (+ 19us) server_negotiation.cc:407] Connection header received
0506 15:42:55.996577 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:55.996582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:55.996625 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:55.996783 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:55.997684 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.998401 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:55.999171 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:55.999994 (+ 823us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.002055 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.002063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.002064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.002069 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.003878 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.004208 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.004211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.004211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.004249 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.004388 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.004388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.004389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.004522 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.004645 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.004745 (+ 100us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.004820 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
I20240506 15:42:56.005950 31949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:42:56 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
May 06 15:42:56 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:56.013510 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.006314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.17.240.253:38205)
0506 15:42:56.006381 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:56.006383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:56.006390 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:42:56.006416 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:56.006417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:56.006424 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:56.006465 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:56.006939 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.007710 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.008421 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.008607 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.010310 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.010314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.010315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.010322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.012345 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.012717 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.012718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.012718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.012738 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.012907 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.012908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.012908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.013034 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.013085 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.013319 (+ 234us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.013411 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240506 15:42:56.013509 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.006196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:38205 (local address 127.0.0.1:52190)
0506 15:42:56.006233 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:56.006243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:56.006358 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:56.006471 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:56.006472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:56.006642 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:56.006841 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:56.006843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.007792 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.007793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:56.008319 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:56.008321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.008393 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.008926 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:56.008930 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:56.010224 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:56.012473 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:56.012474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:56.012474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:56.012631 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:56.012799 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:56.012799 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:56.012800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:56.012846 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:56.013179 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:56.013180 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:56.013221 (+ 41us) client_negotiation.cc:769] Sending connection context
0506 15:42:56.013277 (+ 56us) client_negotiation.cc:240] Negotiation successful
0506 15:42:56.013387 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20240506 15:42:56.013662 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.006086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.17.240.254:39119)
0506 15:42:56.006135 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:56.006137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:56.006147 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:42:56.006168 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:56.006169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:56.006180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:56.006243 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:56.006966 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.007675 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.008416 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.009060 (+ 644us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.010932 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.010936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.010937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.010944 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.012574 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.012916 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.012917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.012917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.012937 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.013166 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.013166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.013167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.013253 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.013415 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.013483 (+ 68us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.013573 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:42:56.014163 31949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:42:56 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
May 06 15:42:56 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
I20240506 15:42:56.020834 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.014472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41744)
0506 15:42:56.014517 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:56.014528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:56.014567 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:56.014698 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:56.014699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:56.014871 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:56.015075 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:56.015077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.015673 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.015674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:56.016007 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:56.016008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.016070 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.016515 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:56.016519 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:56.017986 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:56.019884 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:56.019885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:56.019885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:56.020072 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:56.020229 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:56.020230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:56.020231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:56.020262 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:56.020466 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:56.020466 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:56.020512 (+ 46us) client_negotiation.cc:769] Sending connection context
0506 15:42:56.020592 (+ 80us) client_negotiation.cc:240] Negotiation successful
0506 15:42:56.020696 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20240506 15:42:56.020848 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.014508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.17.240.252:35319)
0506 15:42:56.014558 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:56.014560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:56.014568 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:42:56.014628 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:56.014629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:56.014639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:56.014695 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:56.015149 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.015599 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.016071 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.016582 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.018072 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.018076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.018077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.018084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.019757 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.020148 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.020149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.020150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.020169 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.020320 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.020320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.020320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.020409 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.020456 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.020610 (+ 154us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.020768 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":31}
I20240506 15:42:56.021783 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.014899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.17.240.253:38205)
0506 15:42:56.014954 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:56.014956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:56.014964 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:42:56.014981 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:56.014981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:56.014989 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:56.015107 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:56.015635 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.016042 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.016494 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.016665 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.018845 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.018849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.018849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.018855 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.020751 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.021139 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.021140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.021141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.021166 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.021316 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.021317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.021317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.021434 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.021564 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.021656 (+ 92us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.021723 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240506 15:42:56.023097 31949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67: Bootstrap starting.
I20240506 15:42:56.023077 31799 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } attempt: 2
I20240506 15:42:56.023515 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:42:56.023792 31949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:56.024098 31949 log.cc:824] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:56.024960 31949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67: No bootstrap required, opened a new log
May 06 15:42:56 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
I20240506 15:42:56.026851 31949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } }
I20240506 15:42:56.026993 31949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:56.027069 31949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9503191478014f839deb279978f4ce67, State: Initialized, Role: FOLLOWER
I20240506 15:42:56.027179 31949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } }
I20240506 15:42:56.027686 31949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:42:56.027772 31960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } } }
I20240506 15:42:56.028666 31960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:42:56.029646 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.023301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41752)
0506 15:42:56.023354 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:56.023402 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:56.023457 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:56.023590 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:56.023591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:56.023740 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:56.023924 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:56.023928 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.024616 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.024618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:56.025067 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:56.025070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.025124 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.025595 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:56.025599 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:56.026730 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:56.028655 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:56.028657 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:56.028657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:56.028860 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:56.029032 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:56.029032 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:56.029033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:56.029063 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:56.029329 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:56.029330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:56.029380 (+ 50us) client_negotiation.cc:769] Sending connection context
0506 15:42:56.029443 (+ 63us) client_negotiation.cc:240] Negotiation successful
0506 15:42:56.029551 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
W20240506 15:42:56.029661 31971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:42:56.030026 31971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:42:56.032595 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.025298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.17.240.253:38205)
0506 15:42:56.025373 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:56.025375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:56.025384 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:42:56.025407 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:56.025408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:56.025416 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:56.025472 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:56.026390 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.027056 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.027776 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.027951 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.029514 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.029518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.029518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.029525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.031350 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.031669 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.031670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.031670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.031692 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.031827 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.031827 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.031828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.031923 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.032011 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.032425 (+ 414us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.032502 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240506 15:42:56.030439 31881 master_runner.cc:186] Error getting master registration for 127.17.240.254:39119: OK, instance_id { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" instance_seqno: 1715010175681666 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240506 15:42:56.033198 31881 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 15:42:56.033764 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.023341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.17.240.252:35319)
0506 15:42:56.023372 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:56.023374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:56.023450 (+ 76us) server_negotiation.cc:407] Connection header received
0506 15:42:56.023513 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:56.023514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:56.023524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:56.023988 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:56.024019 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.024518 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.025376 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.026045 (+ 669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.026812 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.026815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.026816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.026821 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.028574 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.028933 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.028934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.028934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.028958 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.029127 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.029127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.029128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.029247 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.029303 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.033592 (+ 4289us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.033681 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240506 15:42:56.035138 31799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab: Bootstrap starting.
I20240506 15:42:56.035851 31799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:56.036142 31799 log.cc:824] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:56.036968 31799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab: No bootstrap required, opened a new log
I20240506 15:42:56.038822 31799 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } }
I20240506 15:42:56.038961 31799 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:56.038987 31799 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5479e35fecb4829aac126b22cce94ab, State: Initialized, Role: FOLLOWER
I20240506 15:42:56.039145 31799 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } }
I20240506 15:42:56.039502 31799 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:42:56.039459 31977 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } } }
I20240506 15:42:56.039683 31977 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:42:56.040585 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.029709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.17.240.252:35319)
0506 15:42:56.030140 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:56.030142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:56.030531 (+ 389us) server_negotiation.cc:407] Connection header received
0506 15:42:56.030553 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:56.030554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:56.030576 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:56.033047 (+ 2471us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:56.033917 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.036174 (+ 2257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.036858 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.037167 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.037924 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.037928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.037928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.037937 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.039514 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.039890 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.039891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.039891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.039912 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.040080 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.040082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.040082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.040205 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.040290 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.040396 (+ 106us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.040477 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":78,"threads_started":1}
I20240506 15:42:56.041730 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.029371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.17.240.254:39119)
0506 15:42:56.029412 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:56.029414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:56.030292 (+ 878us) server_negotiation.cc:407] Connection header received
0506 15:42:56.030397 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:56.030399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:56.030410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:56.030441 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:56.031376 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.032056 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.032813 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.032978 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.034430 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.034433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.034434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.034440 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.036025 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.040944 (+ 4919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.040946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.040946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.040974 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.041158 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.041159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.041159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.041265 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.041333 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.041558 (+ 225us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.041646 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20240506 15:42:56.041766 31988 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:42:56.041812 31988 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:42:56.042388 31731 master_runner.cc:186] Error getting master registration for 127.17.240.253:38205: OK, instance_id { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" instance_seqno: 1715010175832049 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240506 15:42:56.044247 31731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 15:42:56.066848 31874 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } attempt: 3
May 06 15:42:56 dist-test-slave-v0k1 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
I20240506 15:42:56.073433 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.067251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:35319 (local address 127.0.0.1:41770)
0506 15:42:56.067338 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:56.067350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:56.067390 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:56.067648 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:56.067649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:56.067813 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:56.067951 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:56.067952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.068560 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.068561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:56.068905 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:56.068907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.068987 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.069485 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:56.069490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:56.070647 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:56.072391 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:56.072392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:56.072392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:56.072542 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:56.072750 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:56.072751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:56.072752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:56.072803 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:56.073099 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:56.073100 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:56.073148 (+ 48us) client_negotiation.cc:769] Sending connection context
0506 15:42:56.073198 (+ 50us) client_negotiation.cc:240] Negotiation successful
0506 15:42:56.073303 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20240506 15:42:56.073952 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:56.067498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.17.240.252:35319)
0506 15:42:56.067541 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:56.067544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:56.067550 (+ 6us) server_negotiation.cc:407] Connection header received
0506 15:42:56.067568 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:56.067569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:56.067577 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:56.067630 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:56.068031 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.068488 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:56.068991 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:56.069531 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:56.070740 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:56.070743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:56.070744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:56.070750 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:56.072312 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.072605 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.072606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.072606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.072632 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:56.072895 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:56.072896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:56.072896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:56.073021 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:56.073789 (+ 768us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:56.073823 (+ 34us) server_negotiation.cc:299] Negotiation successful
0506 15:42:56.073894 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:42:56.075127 31874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355: Bootstrap starting.
I20240506 15:42:56.075716 31874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:56.075970 31874 log.cc:824] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:56.076663 31874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355: No bootstrap required, opened a new log
I20240506 15:42:56.078071 31874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } }
I20240506 15:42:56.078176 31874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:56.078203 31874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1206a7560fe44da5ae596d3164aba355, State: Initialized, Role: FOLLOWER
I20240506 15:42:56.078322 31874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } }
I20240506 15:42:56.078663 31874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:42:56.078639 31991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } } }
I20240506 15:42:56.078852 31991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [sys.catalog]: This master's current role is: FOLLOWER
W20240506 15:42:56.079784 31827 tablet.cc:2369] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:42:56.080447 32002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:42:56.080498 32002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:42:56.081694 31977 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:42:56.081792 31977 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } }
I20240506 15:42:56.082018 31977 leader_election.cc:288] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1206a7560fe44da5ae596d3164aba355 (127.17.240.253:38205), 9503191478014f839deb279978f4ce67 (127.17.240.252:35319)
I20240506 15:42:56.082449 31848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5479e35fecb4829aac126b22cce94ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1206a7560fe44da5ae596d3164aba355" is_pre_election: true
I20240506 15:42:56.082573 31848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5479e35fecb4829aac126b22cce94ab in term 0.
I20240506 15:42:56.082603 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5479e35fecb4829aac126b22cce94ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9503191478014f839deb279978f4ce67" is_pre_election: true
I20240506 15:42:56.082739 31748 leader_election.cc:302] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1206a7560fe44da5ae596d3164aba355, b5479e35fecb4829aac126b22cce94ab; no voters:
I20240506 15:42:56.082765 31923 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5479e35fecb4829aac126b22cce94ab in term 0.
I20240506 15:42:56.082857 31977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:42:56.082917 31977 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:42:56.082945 31977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:56.083999 31977 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } }
I20240506 15:42:56.084129 31977 leader_election.cc:288] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [CANDIDATE]: Term 1 election: Requested vote from peers 1206a7560fe44da5ae596d3164aba355 (127.17.240.253:38205), 9503191478014f839deb279978f4ce67 (127.17.240.252:35319)
I20240506 15:42:56.084316 31848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5479e35fecb4829aac126b22cce94ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1206a7560fe44da5ae596d3164aba355"
I20240506 15:42:56.084379 31848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:56.084962 31806 master_runner.cc:263] No leader master found from master 1206a7560fe44da5ae596d3164aba355
I20240506 15:42:56.085016 31806 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 15:42:56.085266 31848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5479e35fecb4829aac126b22cce94ab in term 1.
I20240506 15:42:56.085433 31748 leader_election.cc:302] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1206a7560fe44da5ae596d3164aba355, b5479e35fecb4829aac126b22cce94ab; no voters:
I20240506 15:42:56.085554 31977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:42:56.085692 31977 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [term 1 LEADER]: Becoming Leader. State: Replica: b5479e35fecb4829aac126b22cce94ab, State: Running, Role: LEADER
I20240506 15:42:56.085690 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5479e35fecb4829aac126b22cce94ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9503191478014f839deb279978f4ce67"
I20240506 15:42:56.085750 31923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:56.085795 31977 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } }
I20240506 15:42:56.086144 31977 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5479e35fecb4829aac126b22cce94ab. Latest consensus state: current_term: 1 leader_uuid: "b5479e35fecb4829aac126b22cce94ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } } }
I20240506 15:42:56.086217 31977 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:56.086652 31923 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5479e35fecb4829aac126b22cce94ab in term 1.
I20240506 15:42:56.087146 32006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:42:56.087386 32006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:42:56.091897 31848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [term 1 FOLLOWER]: Refusing update from remote peer b5479e35fecb4829aac126b22cce94ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:42:56.091904 31923 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [term 1 FOLLOWER]: Refusing update from remote peer b5479e35fecb4829aac126b22cce94ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:42:56.092186 31977 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:42:56.092315 31977 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:42:56.093503 31960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5479e35fecb4829aac126b22cce94ab. Latest consensus state: current_term: 1 leader_uuid: "b5479e35fecb4829aac126b22cce94ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } } }
I20240506 15:42:56.093603 31960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:42:56.094460 31991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5479e35fecb4829aac126b22cce94ab. Latest consensus state: current_term: 1 leader_uuid: "b5479e35fecb4829aac126b22cce94ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } } }
I20240506 15:42:56.094555 31991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:42:56.095755 31977 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5479e35fecb4829aac126b22cce94ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } } }
I20240506 15:42:56.095841 31977 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:56.096117 31991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5479e35fecb4829aac126b22cce94ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } } }
I20240506 15:42:56.096194 31991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:42:56.096449 32007 mvcc.cc:205] Tried to move back new op lower bound from 7024681681269919744 to 7024681681247948800. Current Snapshot: MvccSnapshot[applied={T|T < 7024681681269919744}]
I20240506 15:42:56.096892 32010 mvcc.cc:205] Tried to move back new op lower bound from 7024681681269919744 to 7024681681247948800. Current Snapshot: MvccSnapshot[applied={T|T < 7024681681269919744}]
I20240506 15:42:56.099920 32006 catalog_manager.cc:1347] Generated new cluster ID: 23699e76ffc445dcb7341539fa7733b6
I20240506 15:42:56.099987 32006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:42:56.100230 31960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5479e35fecb4829aac126b22cce94ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } } }
I20240506 15:42:56.100322 31960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:42:56.099920 31977 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5479e35fecb4829aac126b22cce94ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5479e35fecb4829aac126b22cce94ab" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39119 } } peers { permanent_uuid: "1206a7560fe44da5ae596d3164aba355" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 38205 } } peers { permanent_uuid: "9503191478014f839deb279978f4ce67" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 35319 } } }
I20240506 15:42:56.103212 31977 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:56.103358 32008 mvcc.cc:205] Tried to move back new op lower bound from 7024681681269919744 to 7024681681247948800. Current Snapshot: MvccSnapshot[applied={T|T < 7024681681269919744}]
I20240506 15:42:56.119402 32006 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:42:56.120198 32006 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:42:56.129730 32006 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b5479e35fecb4829aac126b22cce94ab: Generated new TSK 0
I20240506 15:42:56.129974 32006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:42:57.030514 31971 catalog_manager.cc:1259] Loaded cluster ID: 23699e76ffc445dcb7341539fa7733b6
I20240506 15:42:57.030598 31971 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67: loading cluster ID for follower catalog manager: success
I20240506 15:42:57.031558 31971 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67: acquiring CA information for follower catalog manager: success
I20240506 15:42:57.031867 31971 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9503191478014f839deb279978f4ce67: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 15:42:57.080993 32002 catalog_manager.cc:1259] Loaded cluster ID: 23699e76ffc445dcb7341539fa7733b6
I20240506 15:42:57.081079 32002 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355: loading cluster ID for follower catalog manager: success
I20240506 15:42:57.082062 32002 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355: acquiring CA information for follower catalog manager: success
I20240506 15:42:57.082391 32002 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1206a7560fe44da5ae596d3164aba355: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 15:42:57.184470 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 31731
I20240506 15:42:57.189918 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 31806
I20240506 15:42:57.195055 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 31881
2024-05-06T15:42:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2701 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:42:57 dist-test-slave-v0k1 krb5kdc[32024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:42:57 dist-test-slave-v0k1 krb5kdc[32024](info): set up 2 sockets
May 06 15:42:57 dist-test-slave-v0k1 krb5kdc[32024](info): commencing operation
krb5kdc: starting...
W20240506 15:42:58.151285 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:42:58 dist-test-slave-v0k1 krb5kdc[32024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:42:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:42:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:42:58.275866 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36477
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:36477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:58.359313 32037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:58.359539 32037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:58.359565 32037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:58.361142 32037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:42:58.361208 32037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:58.361231 32037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:58.361248 32037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:42:58.361271 32037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:42:58.364032 32037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:36477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:58.364346 32037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:58.364648 32037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:58.370335 32042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:58.370837 32043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:58.371562 32045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:58.372413 32037 server_base.cc:1034] running on GCE node
I20240506 15:42:58.372653 32037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:58.372920 32037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:58.374081 32037 hybrid_clock.cc:648] HybridClock initialized: now 1715010178374057 us; error 37 us; skew 500 ppm
May 06 15:42:58 dist-test-slave-v0k1 krb5kdc[32024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010178, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:58.376636 32037 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:42:58.377063 32037 webserver.cc:460] Webserver started at http://127.17.240.254:40919/ using document root <none> and password file <none>
I20240506 15:42:58.377251 32037 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:58.377297 32037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:58.377419 32037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:42:58.378451 32037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "83a2fcb621d94444a8f06aafc3e90e37"
format_stamp: "Formatted at 2024-05-06 15:42:58 on dist-test-slave-v0k1"
server_key: "436008d0fa23048be1d4d653f52f1085"
server_key_iv: "41c1e1daf0308d054af84fc0d83bcd9a"
server_key_version: "encryptionkey@0"
I20240506 15:42:58.378774 32037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "83a2fcb621d94444a8f06aafc3e90e37"
format_stamp: "Formatted at 2024-05-06 15:42:58 on dist-test-slave-v0k1"
server_key: "436008d0fa23048be1d4d653f52f1085"
server_key_iv: "41c1e1daf0308d054af84fc0d83bcd9a"
server_key_version: "encryptionkey@0"
I20240506 15:42:58.380088 32037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:42:58.380932 32052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:58.381263 32037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:42:58.381363 32037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "83a2fcb621d94444a8f06aafc3e90e37"
format_stamp: "Formatted at 2024-05-06 15:42:58 on dist-test-slave-v0k1"
server_key: "436008d0fa23048be1d4d653f52f1085"
server_key_iv: "41c1e1daf0308d054af84fc0d83bcd9a"
server_key_version: "encryptionkey@0"
I20240506 15:42:58.381439 32037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:58.393971 32037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:58.396234 32037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:58.396369 32037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:58.400359 32037 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:36477
I20240506 15:42:58.400445 32104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:36477 every 8 connection(s)
I20240506 15:42:58.400754 32037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:42:58.401532 32105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:58.403869 32105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37: Bootstrap starting.
I20240506 15:42:58.404486 32105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:58.404769 32105 log.cc:824] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:58.405509 32105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37: No bootstrap required, opened a new log
I20240506 15:42:58.407055 32105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a2fcb621d94444a8f06aafc3e90e37" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36477 } }
I20240506 15:42:58.407195 32105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:58.407212 32105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a2fcb621d94444a8f06aafc3e90e37, State: Initialized, Role: FOLLOWER
I20240506 15:42:58.407341 32105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a2fcb621d94444a8f06aafc3e90e37" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36477 } }
I20240506 15:42:58.407414 32105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:42:58.407440 32105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:42:58.407488 32105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:58.408100 32105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a2fcb621d94444a8f06aafc3e90e37" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36477 } }
I20240506 15:42:58.408193 32105 leader_election.cc:302] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83a2fcb621d94444a8f06aafc3e90e37; no voters:
I20240506 15:42:58.408361 32105 leader_election.cc:288] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:42:58.408617 32108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:42:58.408632 32105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:42:58.408799 32108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [term 1 LEADER]: Becoming Leader. State: Replica: 83a2fcb621d94444a8f06aafc3e90e37, State: Running, Role: LEADER
I20240506 15:42:58.408890 32108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a2fcb621d94444a8f06aafc3e90e37" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36477 } }
I20240506 15:42:58.409242 32109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83a2fcb621d94444a8f06aafc3e90e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a2fcb621d94444a8f06aafc3e90e37" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36477 } } }
I20240506 15:42:58.409334 32109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:58.409667 32110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83a2fcb621d94444a8f06aafc3e90e37. Latest consensus state: current_term: 1 leader_uuid: "83a2fcb621d94444a8f06aafc3e90e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a2fcb621d94444a8f06aafc3e90e37" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36477 } } }
I20240506 15:42:58.409768 32110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37 [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:58.409667 32117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:42:58.410373 32117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:42:58.410878 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 32037
I20240506 15:42:58.410984 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:42:58.411211 18371 external_mini_cluster.cc:1428] Setting key 694a22fad0092ea1cbfefc79df053aaf
I20240506 15:42:58.412554 32117 catalog_manager.cc:1347] Generated new cluster ID: fc16fb738d6e4e6ab4afc568f7ae449f
I20240506 15:42:58.412611 32117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:42:58 dist-test-slave-v0k1 krb5kdc[32024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:58.422736 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:58.411782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.17.240.254:36477)
0506 15:42:58.411961 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:58.411973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:58.412001 (+ 28us) server_negotiation.cc:407] Connection header received
0506 15:42:58.412118 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:58.412123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:58.412168 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:58.412322 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:58.413416 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:58.414616 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:58.415767 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:58.416053 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:58.418521 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:58.418529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:58.418530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:58.418554 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:58.420768 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:58.421184 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:58.421187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:58.421188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:58.421254 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:58.421439 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:58.421440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:58.421440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:58.421704 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:58.421847 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:58.422347 (+ 500us) server_negotiation.cc:299] Negotiation successful
0506 15:42:58.422494 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20240506 15:42:58.423959 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:42:58.432143 32117 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:42:58.432731 32117 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:42:58.445061 32117 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 83a2fcb621d94444a8f06aafc3e90e37: Generated new TSK 0
I20240506 15:42:58.445243 32117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:42:58.478128 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 32037
2024-05-06T15:42:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1281 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:42:58 dist-test-slave-v0k1 krb5kdc[32132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:42:58 dist-test-slave-v0k1 krb5kdc[32132](info): set up 2 sockets
May 06 15:42:58 dist-test-slave-v0k1 krb5kdc[32132](info): commencing operation
krb5kdc: starting...
W20240506 15:42:59.443281 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:42:59 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:42:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:42:59.563638 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38247
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:38247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:59.644007 32145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:59.644217 32145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:59.644238 32145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:59.645524 32145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:42:59.645571 32145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:59.645586 32145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:59.645597 32145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:42:59.645610 32145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:42:59.647104 32145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:38247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:59.647305 32145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:59.647477 32145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:59.652287 32151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:59.652655 32153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:59.653177 32150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:59.654063 32145 server_base.cc:1034] running on GCE node
I20240506 15:42:59.654294 32145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:59.654538 32145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:59.655684 32145 hybrid_clock.cc:648] HybridClock initialized: now 1715010179655673 us; error 24 us; skew 500 ppm
May 06 15:42:59 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010179, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:59.658182 32145 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:42:59.658607 32145 webserver.cc:460] Webserver started at http://127.17.240.254:39109/ using document root <none> and password file <none>
I20240506 15:42:59.658787 32145 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:59.658831 32145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:59.658957 32145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:42:59.659994 32145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "89668ff351f7428fae5622bbc4b3a14c"
format_stamp: "Formatted at 2024-05-06 15:42:59 on dist-test-slave-v0k1"
server_key: "2e0236d2994eb51a82e869db561a8569"
server_key_iv: "ebcc61d358f0c2bd09dfe5c093e97e8a"
server_key_version: "encryptionkey@0"
I20240506 15:42:59.660305 32145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "89668ff351f7428fae5622bbc4b3a14c"
format_stamp: "Formatted at 2024-05-06 15:42:59 on dist-test-slave-v0k1"
server_key: "2e0236d2994eb51a82e869db561a8569"
server_key_iv: "ebcc61d358f0c2bd09dfe5c093e97e8a"
server_key_version: "encryptionkey@0"
I20240506 15:42:59.661526 32145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:42:59.662226 32160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:59.662425 32145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:42:59.662499 32145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "89668ff351f7428fae5622bbc4b3a14c"
format_stamp: "Formatted at 2024-05-06 15:42:59 on dist-test-slave-v0k1"
server_key: "2e0236d2994eb51a82e869db561a8569"
server_key_iv: "ebcc61d358f0c2bd09dfe5c093e97e8a"
server_key_version: "encryptionkey@0"
I20240506 15:42:59.662575 32145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:59.681160 32145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:59.683025 32145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:59.683147 32145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:59.686887 32145 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:38247
I20240506 15:42:59.686964 32212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:38247 every 8 connection(s)
I20240506 15:42:59.687291 32145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:42:59.688143 32213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:42:59.688560 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 32145
I20240506 15:42:59.688692 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:42:59.688887 18371 external_mini_cluster.cc:1428] Setting key 04281cf8b3649f30a8c243f17c30af43
I20240506 15:42:59.691138 32213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Bootstrap starting.
I20240506 15:42:59.691844 32213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Neither blocks nor log segments found. Creating new log.
I20240506 15:42:59.692137 32213 log.cc:824] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Log is configured to *not* fsync() on all Append() calls
I20240506 15:42:59.692961 32213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: No bootstrap required, opened a new log
May 06 15:42:59 dist-test-slave-v0k1 krb5kdc[32132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:59.694933 32213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:42:59.695111 32213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:42:59.695148 32213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89668ff351f7428fae5622bbc4b3a14c, State: Initialized, Role: FOLLOWER
I20240506 15:42:59.695283 32213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:42:59.695345 32213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:42:59.695381 32213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:42:59.695538 32213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:42:59.696286 32213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:42:59.696420 32213 leader_election.cc:302] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89668ff351f7428fae5622bbc4b3a14c; no voters:
I20240506 15:42:59.696617 32213 leader_election.cc:288] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:42:59.696674 32218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:42:59.696879 32213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:42:59.696895 32218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 1 LEADER]: Becoming Leader. State: Replica: 89668ff351f7428fae5622bbc4b3a14c, State: Running, Role: LEADER
I20240506 15:42:59.697130 32218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:42:59.698422 32219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89668ff351f7428fae5622bbc4b3a14c. Latest consensus state: current_term: 1 leader_uuid: "89668ff351f7428fae5622bbc4b3a14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } } }
I20240506 15:42:59.698462 32220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89668ff351f7428fae5622bbc4b3a14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } } }
W20240506 15:42:59.698793 32233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240506 15:42:59.698894 32219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: This master's current role is: LEADER
W20240506 15:42:59.698927 32233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:42:59.698982 32220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: This master's current role is: LEADER
I20240506 15:42:59.699154 32235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:42:59.699349 32235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:42:59.699883 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:59.689486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.17.240.254:38247)
0506 15:42:59.689676 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:59.689688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:59.689713 (+ 25us) server_negotiation.cc:407] Connection header received
0506 15:42:59.690212 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:59.690221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:59.690282 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:59.690474 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:59.691275 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:59.692474 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:59.693167 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:59.693431 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:59.695698 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:59.695706 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:59.695707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:59.695725 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:59.698230 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:59.698654 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:59.698657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:59.698658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:59.698724 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:59.698890 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:59.698892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:59.698892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:59.699234 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:59.699407 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:59.699549 (+ 142us) server_negotiation.cc:299] Negotiation successful
0506 15:42:59.699687 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20240506 15:42:59.700948 32235 catalog_manager.cc:1347] Generated new cluster ID: 898d06095a284549bb45413f1912dbc4
I20240506 15:42:59.701015 32235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:42:59.712419 32235 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:42:59.712985 32235 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:42:59.726634 32235 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Generated new TSK 0
I20240506 15:42:59.726863 32235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:42:59.758107 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38247
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:42:59.842849 32239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:42:59.843106 32239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:42:59.843154 32239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:42:59.844571 32239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:42:59.844628 32239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:42:59.844672 32239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:42:59.846236 32239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:42:59.846487 32239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:42:59.846678 32239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:42:59.852535 32245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:59.852941 32244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:42:59.853541 32247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:42:59.854415 32239 server_base.cc:1034] running on GCE node
I20240506 15:42:59.854619 32239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:42:59.854843 32239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:42:59.856002 32239 hybrid_clock.cc:648] HybridClock initialized: now 1715010179855975 us; error 45 us; skew 500 ppm
May 06 15:42:59 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010179, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:42:59.858901 32239 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:42:59.859462 32239 webserver.cc:460] Webserver started at http://127.17.240.193:40017/ using document root <none> and password file <none>
I20240506 15:42:59.859678 32239 fs_manager.cc:359] Metadata directory not provided
I20240506 15:42:59.859730 32239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:42:59.859836 32239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:42:59.861091 32239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "76ec08e9f561432eacc0cb9bee1164b1"
format_stamp: "Formatted at 2024-05-06 15:42:59 on dist-test-slave-v0k1"
server_key: "c4ed6f8d453046891f6c6d73b1ff4371"
server_key_iv: "a00ee4a6705b7f874fbd72e862998c75"
server_key_version: "encryptionkey@0"
I20240506 15:42:59.861454 32239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "76ec08e9f561432eacc0cb9bee1164b1"
format_stamp: "Formatted at 2024-05-06 15:42:59 on dist-test-slave-v0k1"
server_key: "c4ed6f8d453046891f6c6d73b1ff4371"
server_key_iv: "a00ee4a6705b7f874fbd72e862998c75"
server_key_version: "encryptionkey@0"
I20240506 15:42:59.863041 32239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:42:59.863867 32254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:59.864082 32239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:42:59.864163 32239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "76ec08e9f561432eacc0cb9bee1164b1"
format_stamp: "Formatted at 2024-05-06 15:42:59 on dist-test-slave-v0k1"
server_key: "c4ed6f8d453046891f6c6d73b1ff4371"
server_key_iv: "a00ee4a6705b7f874fbd72e862998c75"
server_key_version: "encryptionkey@0"
I20240506 15:42:59.864223 32239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:42:59.885296 32239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:42:59.887176 32239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:42:59.887281 32239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:42:59.887476 32239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:42:59.887754 32239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:42:59.887786 32239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:59.887809 32239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:42:59.887825 32239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:42:59.893361 32239 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:37011
I20240506 15:42:59.893436 32367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:37011 every 8 connection(s)
I20240506 15:42:59.893700 32239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
May 06 15:42:59 dist-test-slave-v0k1 krb5kdc[32132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010179, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:42:59.901999 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:59.894411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:57157 (local address 127.17.240.254:38247)
0506 15:42:59.894494 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0506 15:42:59.894496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:42:59.894640 (+ 144us) server_negotiation.cc:407] Connection header received
0506 15:42:59.895165 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:42:59.895166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:42:59.895175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:42:59.895231 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:42:59.896268 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:59.896787 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:42:59.897481 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:42:59.897611 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:59.899638 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:42:59.899645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:42:59.899645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:42:59.899652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:42:59.900954 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:59.901322 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:59.901322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:59.901323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:59.901344 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:42:59.901488 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:42:59.901488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:42:59.901489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:42:59.901574 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:42:59.901620 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0506 15:42:59.901845 (+ 225us) server_negotiation.cc:299] Negotiation successful
0506 15:42:59.901929 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240506 15:42:59.902112 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:42:59.894380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:57157)
0506 15:42:59.894535 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0506 15:42:59.894560 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 15:42:59.895073 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:42:59.895288 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:42:59.895292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:42:59.895587 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:42:59.896169 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:59.896173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:59.896892 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:42:59.896893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:42:59.897363 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:42:59.897366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:42:59.897492 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:42:59.898017 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:42:59.898041 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:42:59.899533 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:42:59.901044 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:59.901048 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:59.901051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:59.901256 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:59.901394 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:42:59.901394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:42:59.901395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:42:59.901444 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:42:59.901618 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:42:59.901620 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:42:59.901728 (+ 108us) client_negotiation.cc:769] Sending connection context
0506 15:42:59.901812 (+ 84us) client_negotiation.cc:240] Negotiation successful
0506 15:42:59.901920 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20240506 15:42:59.902674 32368 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38247
I20240506 15:42:59.902767 32368 heartbeater.cc:464] Registering TS with master...
I20240506 15:42:59.902983 32368 heartbeater.cc:511] Master 127.17.240.254:38247 requested a full tablet report, sending...
I20240506 15:42:59.903548 32177 ts_manager.cc:194] Registered new tserver with Master: 76ec08e9f561432eacc0cb9bee1164b1 (127.17.240.193:37011)
I20240506 15:42:59.903766 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 32239
I20240506 15:42:59.903877 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:42:59.904053 18371 external_mini_cluster.cc:1428] Setting key eec745a76f1a6ca3354647599bd5695b
I20240506 15:42:59.904305 32177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:57157
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:42:59.931354 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38247
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:00.011643 32373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:00.011837 32373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:00.011862 32373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:00.013212 32373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:00.013265 32373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:00.013291 32373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:43:00.014787 32373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:00.015002 32373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:00.015228 32373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:00.020177 32379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:00.020902 32378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:00.021641 32381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:00.022177 32373 server_base.cc:1034] running on GCE node
I20240506 15:43:00.022394 32373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:00.022642 32373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:00.023811 32373 hybrid_clock.cc:648] HybridClock initialized: now 1715010180023789 us; error 45 us; skew 500 ppm
May 06 15:43:00 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010180, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:00.026569 32373 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:43:00.027093 32373 webserver.cc:460] Webserver started at http://127.17.240.194:33201/ using document root <none> and password file <none>
I20240506 15:43:00.027284 32373 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:00.027328 32373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:00.027423 32373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:43:00.028450 32373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "19c190e86a9a4160b77226a199fe726d"
format_stamp: "Formatted at 2024-05-06 15:43:00 on dist-test-slave-v0k1"
server_key: "e7ed18e278f9e729fec1432864a3753e"
server_key_iv: "292d02e8d3ad5dca1fbc91948ce1c1c3"
server_key_version: "encryptionkey@0"
I20240506 15:43:00.028759 32373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "19c190e86a9a4160b77226a199fe726d"
format_stamp: "Formatted at 2024-05-06 15:43:00 on dist-test-slave-v0k1"
server_key: "e7ed18e278f9e729fec1432864a3753e"
server_key_iv: "292d02e8d3ad5dca1fbc91948ce1c1c3"
server_key_version: "encryptionkey@0"
I20240506 15:43:00.029978 32373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:43:00.030673 32388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:00.030894 32373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:43:00.030984 32373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "19c190e86a9a4160b77226a199fe726d"
format_stamp: "Formatted at 2024-05-06 15:43:00 on dist-test-slave-v0k1"
server_key: "e7ed18e278f9e729fec1432864a3753e"
server_key_iv: "292d02e8d3ad5dca1fbc91948ce1c1c3"
server_key_version: "encryptionkey@0"
I20240506 15:43:00.031091 32373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:00.040343 32373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:00.042237 32373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:00.042352 32373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:00.042567 32373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:00.042908 32373 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:00.042944 32373 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:00.042979 32373 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:00.043053 32373 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:00.048635 32373 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:36209
I20240506 15:43:00.048688 32501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:36209 every 8 connection(s)
I20240506 15:43:00.049032 32373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
May 06 15:43:00 dist-test-slave-v0k1 krb5kdc[32132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010180, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:00.056427 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 32373
I20240506 15:43:00.056560 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:43:00.056756 18371 external_mini_cluster.cc:1428] Setting key cdc732c852d3cd03d4eb69024e895f14
I20240506 15:43:00.057363 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.049739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:56989 (local address 127.17.240.254:38247)
0506 15:43:00.049782 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:00.049784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:00.049980 (+ 196us) server_negotiation.cc:407] Connection header received
0506 15:43:00.050474 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:00.050475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:00.050487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:00.050548 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:00.051612 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.052095 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.052611 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.052778 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.054634 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:00.054641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:00.054641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:00.054647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:00.056025 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:00.056393 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:00.056394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:00.056395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:00.056426 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:00.056601 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:00.056602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:00.056602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:00.056717 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:00.056777 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:00.057183 (+ 406us) server_negotiation.cc:299] Negotiation successful
0506 15:43:00.057271 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240506 15:43:00.057482 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.049718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:56989)
0506 15:43:00.049864 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:00.049889 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:00.050370 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:00.050611 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:00.050614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:00.050896 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:00.051517 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.051523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.052175 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.052176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:00.052539 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.052540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.052673 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.053211 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:00.053225 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:00.054546 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:00.056104 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:00.056109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:00.056112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:00.056329 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:00.056485 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:00.056486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:00.056486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:00.056541 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:00.056774 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:00.056779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:00.057023 (+ 244us) client_negotiation.cc:769] Sending connection context
0506 15:43:00.057152 (+ 129us) client_negotiation.cc:240] Negotiation successful
0506 15:43:00.057290 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20240506 15:43:00.057897 32502 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38247
I20240506 15:43:00.057986 32502 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:00.058166 32502 heartbeater.cc:511] Master 127.17.240.254:38247 requested a full tablet report, sending...
I20240506 15:43:00.058543 32176 ts_manager.cc:194] Registered new tserver with Master: 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
I20240506 15:43:00.058959 32176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:56989
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:43:00.082670 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38247
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:00.162827 32507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:00.163053 32507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:00.163089 32507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:00.164475 32507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:00.164525 32507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:00.164551 32507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:43:00.166139 32507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:00.166389 32507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:00.166595 32507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:00.171507 32515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:00.172019 32512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:00.172758 32513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:00.173514 32507 server_base.cc:1034] running on GCE node
I20240506 15:43:00.173727 32507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:00.173965 32507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:00.175087 32507 hybrid_clock.cc:648] HybridClock initialized: now 1715010180175066 us; error 30 us; skew 500 ppm
May 06 15:43:00 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010180, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:00.177649 32507 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:43:00.178050 32507 webserver.cc:460] Webserver started at http://127.17.240.195:46663/ using document root <none> and password file <none>
I20240506 15:43:00.178243 32507 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:00.178284 32507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:00.178397 32507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:43:00.179441 32507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "4fae4ef17fbb44bf8dc41d32806c42d9"
format_stamp: "Formatted at 2024-05-06 15:43:00 on dist-test-slave-v0k1"
server_key: "176d30161cf6e15808a5b9fc6f496973"
server_key_iv: "c4e4e413b9d937d9ac727d4da605fe46"
server_key_version: "encryptionkey@0"
I20240506 15:43:00.179737 32507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "4fae4ef17fbb44bf8dc41d32806c42d9"
format_stamp: "Formatted at 2024-05-06 15:43:00 on dist-test-slave-v0k1"
server_key: "176d30161cf6e15808a5b9fc6f496973"
server_key_iv: "c4e4e413b9d937d9ac727d4da605fe46"
server_key_version: "encryptionkey@0"
I20240506 15:43:00.180930 32507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:43:00.181699 32522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:00.181881 32507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:00.181951 32507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "4fae4ef17fbb44bf8dc41d32806c42d9"
format_stamp: "Formatted at 2024-05-06 15:43:00 on dist-test-slave-v0k1"
server_key: "176d30161cf6e15808a5b9fc6f496973"
server_key_iv: "c4e4e413b9d937d9ac727d4da605fe46"
server_key_version: "encryptionkey@0"
I20240506 15:43:00.182022 32507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:00.202275 32507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:00.204639 32507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:00.204764 32507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:00.205017 32507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:00.205334 32507 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:00.205368 32507 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:00.205394 32507 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:00.205406 32507 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:00.211189 32507 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:37101
I20240506 15:43:00.211248 32635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:37101 every 8 connection(s)
I20240506 15:43:00.211553 32507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
May 06 15:43:00 dist-test-slave-v0k1 krb5kdc[32132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010180, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:00.217414 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 32507
I20240506 15:43:00.217506 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:43:00.217715 18371 external_mini_cluster.cc:1428] Setting key 3d471a3c36dccb72228f93d645634359
I20240506 15:43:00.219833 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.212228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:56291 (local address 127.17.240.254:38247)
0506 15:43:00.212288 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:00.212290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:00.212427 (+ 137us) server_negotiation.cc:407] Connection header received
0506 15:43:00.212906 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:00.212907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:00.212916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:00.212982 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:00.214048 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.214554 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.215143 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.215261 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.217329 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:00.217334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:00.217334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:00.217340 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:00.218746 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:00.219160 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:00.219161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:00.219161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:00.219183 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:00.219326 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:00.219327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:00.219327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:00.219432 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:00.219479 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:00.219701 (+ 222us) server_negotiation.cc:299] Negotiation successful
0506 15:43:00.219768 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240506 15:43:00.219956 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.212190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:56291)
0506 15:43:00.212323 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:00.212354 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:00.212808 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:00.213070 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:00.213074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:00.213380 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:00.213960 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.213964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.214635 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.214636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:00.215073 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.215076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.215181 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.215726 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:00.215739 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:00.217239 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:00.218833 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:00.218838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:00.218840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:00.219100 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:00.219237 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:00.219238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:00.219238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:00.219288 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:00.219478 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:00.219482 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:00.219590 (+ 108us) client_negotiation.cc:769] Sending connection context
0506 15:43:00.219678 (+ 88us) client_negotiation.cc:240] Negotiation successful
0506 15:43:00.219782 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20240506 15:43:00.220414 32636 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38247
I20240506 15:43:00.220507 32636 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:00.220688 32636 heartbeater.cc:511] Master 127.17.240.254:38247 requested a full tablet report, sending...
I20240506 15:43:00.221020 32176 ts_manager.cc:194] Registered new tserver with Master: 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101)
I20240506 15:43:00.221408 32176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:56291
I20240506 15:43:00.229635 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:43:00.236804 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.230909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.17.240.254:38247)
0506 15:43:00.230966 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:00.230968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:00.231060 (+ 92us) server_negotiation.cc:407] Connection header received
0506 15:43:00.231133 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:00.231134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:00.231143 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:00.231185 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:00.231895 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.232401 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.232990 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.233147 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.234253 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:00.234256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:00.234257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:00.234263 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:00.235734 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:00.236040 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:00.236041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:00.236041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:00.236059 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:00.236299 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:00.236300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:00.236300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:00.236443 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:00.236524 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:00.236660 (+ 136us) server_negotiation.cc:299] Negotiation successful
0506 15:43:00.236725 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240506 15:43:00.238512 32176 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:43:00.240017 32176 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:43:00.245501 32650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.242496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:36209 (local address 127.0.0.1:35054)
0506 15:43:00.242609 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:00.242616 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:00.242657 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:00.243082 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:00.243083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:00.243084 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:00.243272 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.243273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.244642 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.244643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:00.245218 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.245220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.245259 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.245302 (+ 43us) client_negotiation.cc:769] Sending connection context
0506 15:43:00.245352 (+ 50us) client_negotiation.cc:240] Negotiation successful
0506 15:43:00.245392 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20240506 15:43:00.245633 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.242204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:37101 (local address 127.0.0.1:46834)
0506 15:43:00.242309 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:00.242335 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:00.242458 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:00.243055 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:00.243056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:00.243059 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:00.243257 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.243259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.244623 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.244624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:00.245424 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.245427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.245495 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.245498 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:43:00.245529 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:43:00.245569 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":61,"threads_started":1}
I20240506 15:43:00.245802 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.242258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37011 (local address 127.0.0.1:41224)
0506 15:43:00.242400 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:00.242411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:00.242458 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:00.242909 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:00.242912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:00.242915 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:00.243220 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.243222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.244371 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.244372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:00.245225 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.245227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.245299 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.245303 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:43:00.245353 (+ 50us) client_negotiation.cc:240] Negotiation successful
0506 15:43:00.245391 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":105,"threads_started":1}
I20240506 15:43:00.246348 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.242368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.17.240.195:37101)
0506 15:43:00.242721 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:00.242730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:00.242741 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:43:00.242781 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:00.242783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:00.242799 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:00.242909 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:00.243337 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.244544 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.245815 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.246198 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.246239 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:00.246266 (+ 27us) server_negotiation.cc:299] Negotiation successful
0506 15:43:00.246293 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":89,"threads_started":1}
I20240506 15:43:00.246548 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.242535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.17.240.193:37011)
0506 15:43:00.242713 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:00.242721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:00.242730 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:43:00.242775 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:00.242777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:00.242800 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:00.242895 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:00.243310 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.244289 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.245340 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.246344 (+ 1004us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.246400 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:00.246429 (+ 29us) server_negotiation.cc:299] Negotiation successful
0506 15:43:00.246468 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20240506 15:43:00.246551 32652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.242696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.17.240.194:36209)
0506 15:43:00.242892 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:00.242899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:00.242909 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:43:00.242945 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:00.242947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:00.242963 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:00.243084 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:00.243359 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.244538 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.245765 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.246349 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.246388 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:00.246421 (+ 33us) server_negotiation.cc:299] Negotiation successful
0506 15:43:00.246462 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20240506 15:43:00.246913 32570 tablet_service.cc:1467] Processing CreateTablet for tablet 90da4040f73e4e549f56c1e3c121cbe9 (DEFAULT_TABLE table=test-table [id=abc175982b4a43f5b296c67e948b2c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:00.247079 32436 tablet_service.cc:1467] Processing CreateTablet for tablet 90da4040f73e4e549f56c1e3c121cbe9 (DEFAULT_TABLE table=test-table [id=abc175982b4a43f5b296c67e948b2c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:00.247079 32302 tablet_service.cc:1467] Processing CreateTablet for tablet 90da4040f73e4e549f56c1e3c121cbe9 (DEFAULT_TABLE table=test-table [id=abc175982b4a43f5b296c67e948b2c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:00.247274 32570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90da4040f73e4e549f56c1e3c121cbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:00.247351 32436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90da4040f73e4e549f56c1e3c121cbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:00.247351 32302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90da4040f73e4e549f56c1e3c121cbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:00.249975 32653 tablet_bootstrap.cc:492] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: Bootstrap starting.
I20240506 15:43:00.250392 32655 tablet_bootstrap.cc:492] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: Bootstrap starting.
I20240506 15:43:00.250787 32653 tablet_bootstrap.cc:654] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:00.250804 32654 tablet_bootstrap.cc:492] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Bootstrap starting.
I20240506 15:43:00.250965 32655 tablet_bootstrap.cc:654] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:00.251257 32653 log.cc:824] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:00.251288 32655 log.cc:824] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:00.251381 32654 tablet_bootstrap.cc:654] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:00.251646 32654 log.cc:824] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:00.252043 32655 tablet_bootstrap.cc:492] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: No bootstrap required, opened a new log
I20240506 15:43:00.252116 32655 ts_tablet_manager.cc:1367] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:43:00.252163 32653 tablet_bootstrap.cc:492] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: No bootstrap required, opened a new log
I20240506 15:43:00.252229 32653 ts_tablet_manager.cc:1367] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:43:00.252317 32654 tablet_bootstrap.cc:492] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: No bootstrap required, opened a new log
I20240506 15:43:00.252385 32654 ts_tablet_manager.cc:1367] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:43:00.253983 32654 raft_consensus.cc:348] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:00.254107 32654 raft_consensus.cc:374] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:00.254137 32654 raft_consensus.cc:729] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ec08e9f561432eacc0cb9bee1164b1, State: Initialized, Role: FOLLOWER
I20240506 15:43:00.254261 32654 consensus_queue.cc:260] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:00.254415 32653 raft_consensus.cc:348] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:00.254556 32653 raft_consensus.cc:374] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:00.254559 32654 ts_tablet_manager.cc:1398] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 15:43:00.254585 32653 raft_consensus.cc:729] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fae4ef17fbb44bf8dc41d32806c42d9, State: Initialized, Role: FOLLOWER
I20240506 15:43:00.254669 32368 heartbeater.cc:503] Master 127.17.240.254:38247 was elected leader, sending a full tablet report...
I20240506 15:43:00.254674 32653 consensus_queue.cc:260] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:00.254710 32655 raft_consensus.cc:348] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:00.254839 32655 raft_consensus.cc:374] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:00.254875 32655 raft_consensus.cc:729] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c190e86a9a4160b77226a199fe726d, State: Initialized, Role: FOLLOWER
I20240506 15:43:00.254936 32653 ts_tablet_manager.cc:1398] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:43:00.254984 32655 consensus_queue.cc:260] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:00.255092 32636 heartbeater.cc:503] Master 127.17.240.254:38247 was elected leader, sending a full tablet report...
I20240506 15:43:00.255256 32655 ts_tablet_manager.cc:1398] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240506 15:43:00.255298 32502 heartbeater.cc:503] Master 127.17.240.254:38247 was elected leader, sending a full tablet report...
W20240506 15:43:00.299553 32503 tablet.cc:2369] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:43:00.394471 32369 tablet.cc:2369] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:43:00.462116 32637 tablet.cc:2369] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:43:00.602514 32660 raft_consensus.cc:482] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:43:00.602645 32660 raft_consensus.cc:504] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:00.602964 32660 leader_election.cc:288] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76ec08e9f561432eacc0cb9bee1164b1 (127.17.240.193:37011), 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
I20240506 15:43:00.605361 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.603194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37011 (local address 127.17.240.195:50407)
0506 15:43:00.603280 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:00.603293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:00.603368 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:00.603599 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:00.603599 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:00.603601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:00.603826 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.603828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.604522 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.604523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:00.605183 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.605185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.605243 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.605247 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:43:00.605264 (+ 17us) client_negotiation.cc:240] Negotiation successful
0506 15:43:00.605290 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20240506 15:43:00.605602 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.603278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:36209 (local address 127.17.240.195:53317)
0506 15:43:00.603463 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:00.603473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:00.603546 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:00.603704 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:00.603705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:00.603706 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:00.603913 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.603915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.604733 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.604734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:00.605335 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:00.605337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.605447 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.605452 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:43:00.605483 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:43:00.605517 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":108,"threads_started":1}
I20240506 15:43:00.605899 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.603390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:50407 (local address 127.17.240.193:37011)
0506 15:43:00.603459 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:00.603461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:00.603473 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:43:00.603505 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:00.603506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:00.603517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:00.603577 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:00.603917 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.604418 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.605288 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.605777 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.605788 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:00.605811 (+ 23us) server_negotiation.cc:299] Negotiation successful
0506 15:43:00.605835 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:43:00.606211 32322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90da4040f73e4e549f56c1e3c121cbe9" candidate_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ec08e9f561432eacc0cb9bee1164b1" is_pre_election: true
I20240506 15:43:00.606261 32652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.603428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:53317 (local address 127.17.240.194:36209)
0506 15:43:00.603471 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:00.603480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:00.603547 (+ 67us) server_negotiation.cc:407] Connection header received
0506 15:43:00.603619 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:00.603620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:00.603629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:00.603695 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:00.604028 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.604628 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.605583 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.606096 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.606148 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:00.606175 (+ 27us) server_negotiation.cc:299] Negotiation successful
0506 15:43:00.606211 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:43:00.606372 32322 raft_consensus.cc:2454] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fae4ef17fbb44bf8dc41d32806c42d9 in term 0.
I20240506 15:43:00.606544 32456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90da4040f73e4e549f56c1e3c121cbe9" candidate_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c190e86a9a4160b77226a199fe726d" is_pre_election: true
I20240506 15:43:00.606673 32456 raft_consensus.cc:2454] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fae4ef17fbb44bf8dc41d32806c42d9 in term 0.
I20240506 15:43:00.606846 32525 leader_election.cc:302] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fae4ef17fbb44bf8dc41d32806c42d9, 76ec08e9f561432eacc0cb9bee1164b1; no voters:
I20240506 15:43:00.606971 32660 raft_consensus.cc:2781] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:43:00.607061 32660 raft_consensus.cc:482] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:43:00.607082 32660 raft_consensus.cc:3037] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:00.607738 32660 raft_consensus.cc:504] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:00.607870 32660 leader_election.cc:288] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [CANDIDATE]: Term 1 election: Requested vote from peers 76ec08e9f561432eacc0cb9bee1164b1 (127.17.240.193:37011), 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
I20240506 15:43:00.608098 32456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90da4040f73e4e549f56c1e3c121cbe9" candidate_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c190e86a9a4160b77226a199fe726d"
I20240506 15:43:00.608057 32322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90da4040f73e4e549f56c1e3c121cbe9" candidate_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ec08e9f561432eacc0cb9bee1164b1"
I20240506 15:43:00.608168 32456 raft_consensus.cc:3037] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:00.608206 32322 raft_consensus.cc:3037] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:00.608794 32456 raft_consensus.cc:2454] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fae4ef17fbb44bf8dc41d32806c42d9 in term 1.
I20240506 15:43:00.608964 32322 raft_consensus.cc:2454] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fae4ef17fbb44bf8dc41d32806c42d9 in term 1.
I20240506 15:43:00.609002 32525 leader_election.cc:302] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19c190e86a9a4160b77226a199fe726d, 4fae4ef17fbb44bf8dc41d32806c42d9; no voters:
I20240506 15:43:00.609083 32660 raft_consensus.cc:2781] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:43:00.609280 32660 raft_consensus.cc:686] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 1 LEADER]: Becoming Leader. State: Replica: 4fae4ef17fbb44bf8dc41d32806c42d9, State: Running, Role: LEADER
I20240506 15:43:00.609390 32660 consensus_queue.cc:237] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:00.610191 32177 catalog_manager.cc:5484] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195). New cstate: current_term: 1 leader_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } health_report { overall_health: UNKNOWN } } }
I20240506 15:43:00.688261 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.685986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.17.240.195:37101)
0506 15:43:00.686042 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:00.686044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:00.686053 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:43:00.686096 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:00.686096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:00.686112 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:00.686182 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:43:00.686492 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.686953 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:00.687789 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:00.687909 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:00.687944 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:43:00.688024 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:43:00.688086 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:00.688174 (+ 88us) server_negotiation.cc:299] Negotiation successful
0506 15:43:00.688193 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240506 15:43:00.690047 32322 raft_consensus.cc:1261] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Refusing update from remote peer 4fae4ef17fbb44bf8dc41d32806c42d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:43:00.690057 32456 raft_consensus.cc:1261] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d [term 1 FOLLOWER]: Refusing update from remote peer 4fae4ef17fbb44bf8dc41d32806c42d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:43:00.690376 32660 consensus_queue.cc:1035] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:43:00.690492 32663 consensus_queue.cc:1035] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:43:00.694883 32177 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40476:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:43:00.694983 32177 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40476:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:43:00.695708 32177 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Sending DeleteTablet for 3 replicas of tablet 90da4040f73e4e549f56c1e3c121cbe9
I20240506 15:43:00.696035 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 32239
I20240506 15:43:00.696055 32436 tablet_service.cc:1514] Processing DeleteTablet for tablet 90da4040f73e4e549f56c1e3c121cbe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:00 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:35054
I20240506 15:43:00.696061 32302 tablet_service.cc:1514] Processing DeleteTablet for tablet 90da4040f73e4e549f56c1e3c121cbe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:00 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:41224
I20240506 15:43:00.696727 32570 tablet_service.cc:1514] Processing DeleteTablet for tablet 90da4040f73e4e549f56c1e3c121cbe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:00 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:46834
I20240506 15:43:00.696931 32676 tablet_replica.cc:330] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: stopping tablet replica
I20240506 15:43:00.696916 32677 tablet_replica.cc:330] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: stopping tablet replica
I20240506 15:43:00.697016 32676 raft_consensus.cc:2229] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:43:00.697023 32677 raft_consensus.cc:2229] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:43:00.697099 32676 raft_consensus.cc:2258] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:00.697105 32677 raft_consensus.cc:2258] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:00.697485 32676 ts_tablet_manager.cc:1875] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:43:00.698197 32677 ts_tablet_manager.cc:1875] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:43:00.699466 32676 ts_tablet_manager.cc:1888] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:43:00.699537 32676 log.cc:1196] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/90da4040f73e4e549f56c1e3c121cbe9
I20240506 15:43:00.699573 32677 ts_tablet_manager.cc:1888] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:43:00.699645 32677 log.cc:1196] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/90da4040f73e4e549f56c1e3c121cbe9
I20240506 15:43:00.699826 32676 ts_tablet_manager.cc:1909] T 90da4040f73e4e549f56c1e3c121cbe9 P 19c190e86a9a4160b77226a199fe726d: Deleting consensus metadata
I20240506 15:43:00.699939 32677 ts_tablet_manager.cc:1909] T 90da4040f73e4e549f56c1e3c121cbe9 P 4fae4ef17fbb44bf8dc41d32806c42d9: Deleting consensus metadata
I20240506 15:43:00.700290 32162 catalog_manager.cc:4830] TS 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101): tablet 90da4040f73e4e549f56c1e3c121cbe9 (table test-table [id=abc175982b4a43f5b296c67e948b2c69]) successfully deleted
I20240506 15:43:00.700299 32163 catalog_manager.cc:4830] TS 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209): tablet 90da4040f73e4e549f56c1e3c121cbe9 (table test-table [id=abc175982b4a43f5b296c67e948b2c69]) successfully deleted
W20240506 15:43:00.703590 32163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:37011 (error 108)
I20240506 15:43:00.704054 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 32373
I20240506 15:43:00.704104 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.703958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37011 (local address 127.0.0.1:41240)
0506 15:43:00.704016 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:00.704065 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:37011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20240506 15:43:00.706658 32163 catalog_manager.cc:4573] TS 76ec08e9f561432eacc0cb9bee1164b1 (127.17.240.193:37011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90da4040f73e4e549f56c1e3c121cbe9: Network error: Client connection negotiation failed: client connection to 127.17.240.193:37011: connect: Connection refused (error 111)
I20240506 15:43:00.710108 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 32507
I20240506 15:43:00.714972 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 32145
I20240506 15:43:00.723318 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38247
--webserver_interface=127.17.240.254
--webserver_port=39109
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:38247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:00.802804 32679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:00.803033 32679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:00.803072 32679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:00.804383 32679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:43:00.804431 32679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:00.804457 32679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:00.804476 32679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:43:00.804510 32679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:43:00.806000 32679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:38247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=39109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:00.806245 32679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:00.806434 32679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:00.814100 32688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:00.814536 32686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:00.815119 32685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:00.815955 32679 server_base.cc:1034] running on GCE node
I20240506 15:43:00.816207 32679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:00.816498 32679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:00.817642 32679 hybrid_clock.cc:648] HybridClock initialized: now 1715010180817624 us; error 37 us; skew 500 ppm
May 06 15:43:00 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010180, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:00.824612 32679 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:43:00.825037 32679 webserver.cc:460] Webserver started at http://127.17.240.254:39109/ using document root <none> and password file <none>
I20240506 15:43:00.825240 32679 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:00.825294 32679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:00.826761 32679 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:00.827603 32695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:00.827857 32679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:00.827952 32679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "89668ff351f7428fae5622bbc4b3a14c"
format_stamp: "Formatted at 2024-05-06 15:42:59 on dist-test-slave-v0k1"
server_key: "2e0236d2994eb51a82e869db561a8569"
server_key_iv: "ebcc61d358f0c2bd09dfe5c093e97e8a"
server_key_version: "encryptionkey@0"
I20240506 15:43:00.828305 32679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:00.841293 32679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:00.843092 32679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:00.843219 32679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:00.846980 32679 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:38247
I20240506 15:43:00.847048 32747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:38247 every 8 connection(s)
I20240506 15:43:00.847404 32679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:43:00.847679 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 32679
I20240506 15:43:00.847970 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:37011
--local_ip_for_outbound_sockets=127.17.240.193
--tserver_master_addrs=127.17.240.254:38247
--webserver_port=40017
--webserver_interface=127.17.240.193
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:43:00.848938 32748 sys_catalog.cc:263] Verifying existing consensus state
I20240506 15:43:00.849529 32748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Bootstrap starting.
I20240506 15:43:00.858685 32748 log.cc:824] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:00.860554 32748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:43:00.860839 32748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Bootstrap complete.
I20240506 15:43:00.863047 32748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:43:00.863261 32748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89668ff351f7428fae5622bbc4b3a14c, State: Initialized, Role: FOLLOWER
I20240506 15:43:00.863387 32748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:43:00.863467 32748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:43:00.863514 32748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:43:00.863564 32748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:43:00.864449 32748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:43:00.864543 32748 leader_election.cc:302] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89668ff351f7428fae5622bbc4b3a14c; no voters:
I20240506 15:43:00.864710 32748 leader_election.cc:288] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [CANDIDATE]: Term 2 election: Requested vote from peers
I20240506 15:43:00.864785 32752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:43:00.864923 32748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:43:00.865025 32752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 2 LEADER]: Becoming Leader. State: Replica: 89668ff351f7428fae5622bbc4b3a14c, State: Running, Role: LEADER
I20240506 15:43:00.865114 32752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:43:00.865365 32752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89668ff351f7428fae5622bbc4b3a14c. Latest consensus state: current_term: 2 leader_uuid: "89668ff351f7428fae5622bbc4b3a14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } } }
I20240506 15:43:00.865432 32752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: This master's current role is: LEADER
I20240506 15:43:00.865388 32753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "89668ff351f7428fae5622bbc4b3a14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } } }
I20240506 15:43:00.865649 32753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: This master's current role is: LEADER
I20240506 15:43:00.865599 32759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:43:00.866482 32759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:43:00.866647 32759 catalog_manager.cc:1259] Loaded cluster ID: 898d06095a284549bb45413f1912dbc4
I20240506 15:43:00.866842 32759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:43:00.867100 32759 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20240506 15:43:00.868607 32759 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20240506 15:43:00.870246 32759 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:43:00.870420 32759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Loaded TSK: 0
I20240506 15:43:00.870594 32759 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240506 15:43:00.954684 32750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:00.954934 32750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:00.954977 32750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:00.956957 32750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:00.957013 32750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:00.957055 32750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:43:00.959225 32750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:37011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=40017
--tserver_master_addrs=127.17.240.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:00.959475 32750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:00.959681 32750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:00.964993 310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:00.965660 307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:00.968341 308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:00.969007 32750 server_base.cc:1034] running on GCE node
I20240506 15:43:00.969208 32750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:00.969535 32750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:00.970690 32750 hybrid_clock.cc:648] HybridClock initialized: now 1715010180970667 us; error 44 us; skew 500 ppm
May 06 15:43:00 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010180, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:00.973165 32750 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:43:00.973596 32750 webserver.cc:460] Webserver started at http://127.17.240.193:40017/ using document root <none> and password file <none>
I20240506 15:43:00.973788 32750 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:00.973842 32750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:00.975081 32750 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:00.975666 317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:00.975807 32750 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:43:00.975883 32750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "76ec08e9f561432eacc0cb9bee1164b1"
format_stamp: "Formatted at 2024-05-06 15:42:59 on dist-test-slave-v0k1"
server_key: "c4ed6f8d453046891f6c6d73b1ff4371"
server_key_iv: "a00ee4a6705b7f874fbd72e862998c75"
server_key_version: "encryptionkey@0"
I20240506 15:43:00.976179 32750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:00.987322 32750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:00.989456 32750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:00.989557 32750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:00.989759 32750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:00.990151 324 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240506 15:43:00.991251 32750 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 15:43:00.991322 32750 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:00.991361 32750 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 15:43:00.991950 32750 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 15:43:00.991982 32750 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:00.992067 324 tablet_bootstrap.cc:492] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Bootstrap starting.
I20240506 15:43:00.998545 32750 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:37011
I20240506 15:43:00.998699 431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:37011 every 8 connection(s)
I20240506 15:43:00.998919 32750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:43:01.003305 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 32750
I20240506 15:43:01.003645 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:36209
--local_ip_for_outbound_sockets=127.17.240.194
--tserver_master_addrs=127.17.240.254:38247
--webserver_port=33201
--webserver_interface=127.17.240.194
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:43:01.004880 324 log.cc:824] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Log is configured to *not* fsync() on all Append() calls
May 06 15:43:01 dist-test-slave-v0k1 krb5kdc[32132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010180, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:01.009696 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.999636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:52873)
0506 15:43:00.999834 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:00.999861 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:01.000313 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:01.000924 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:01.000928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:01.001238 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:01.001548 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.001553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.002569 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.002570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:01.002961 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.002962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.003126 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.003992 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:01.004016 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:01.005868 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:01.008374 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:01.008379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:01.008381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:01.008576 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:01.008772 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:01.008772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:01.008773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:01.008827 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:01.009132 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:01.009136 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:01.009246 (+ 110us) client_negotiation.cc:769] Sending connection context
0506 15:43:01.009342 (+ 96us) client_negotiation.cc:240] Negotiation successful
0506 15:43:01.009511 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20240506 15:43:01.009692 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:00.999712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:52873 (local address 127.17.240.254:38247)
0506 15:43:01.000594 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.000602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.000620 (+ 18us) server_negotiation.cc:407] Connection header received
0506 15:43:01.000722 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.000726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.000758 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.000872 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:01.001679 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.002478 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.003086 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.005545 (+ 2459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.005987 (+ 442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:01.006011 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:01.006011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:01.006030 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:01.008289 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:01.008653 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:01.008655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:01.008656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:01.008711 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:01.008884 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:01.008885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:01.008885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:01.009070 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:01.009149 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.009398 (+ 249us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.009514 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":83,"threads_started":1}
I20240506 15:43:01.010587 432 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38247
I20240506 15:43:01.010679 432 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:01.010844 324 tablet_bootstrap.cc:492] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:43:01.010885 432 heartbeater.cc:511] Master 127.17.240.254:38247 requested a full tablet report, sending...
I20240506 15:43:01.011214 324 tablet_bootstrap.cc:492] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Bootstrap complete.
I20240506 15:43:01.011595 32712 ts_manager.cc:194] Registered new tserver with Master: 76ec08e9f561432eacc0cb9bee1164b1 (127.17.240.193:37011)
I20240506 15:43:01.012403 324 ts_tablet_manager.cc:1367] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.006s
I20240506 15:43:01.012696 32712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:52873
I20240506 15:43:01.014055 324 raft_consensus.cc:348] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:01.014417 324 raft_consensus.cc:729] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ec08e9f561432eacc0cb9bee1164b1, State: Initialized, Role: FOLLOWER
I20240506 15:43:01.014551 324 consensus_queue.cc:260] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } }
I20240506 15:43:01.014864 324 ts_tablet_manager.cc:1398] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 15:43:01.014870 432 heartbeater.cc:503] Master 127.17.240.254:38247 was elected leader, sending a full tablet report...
I20240506 15:43:01.018990 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.012113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37011 (local address 127.0.0.1:41248)
0506 15:43:01.015061 (+ 2948us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.015090 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:01.015172 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:01.015501 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:01.015503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:01.015506 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:01.015840 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.015843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.017228 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.017230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:01.018704 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.018707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.018790 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.018816 (+ 26us) client_negotiation.cc:769] Sending connection context
0506 15:43:01.018867 (+ 51us) client_negotiation.cc:240] Negotiation successful
0506 15:43:01.018907 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2915,"thread_start_us":96,"threads_started":1}
I20240506 15:43:01.019559 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.012340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.17.240.193:37011)
0506 15:43:01.015309 (+ 2969us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.015321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.015331 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:43:01.015365 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.015367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.015383 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.015934 (+ 551us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:01.015994 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.017126 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.018804 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.019352 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.019399 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.019432 (+ 33us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.019473 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2937,"thread_start_us":73,"threads_started":1}
I20240506 15:43:01.019853 366 tablet_service.cc:1514] Processing DeleteTablet for tablet 90da4040f73e4e549f56c1e3c121cbe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:00 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:41248
I20240506 15:43:01.020445 441 tablet_replica.cc:330] stopping tablet replica
I20240506 15:43:01.020538 441 raft_consensus.cc:2229] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:43:01.020607 441 raft_consensus.cc:2258] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:01.020954 441 ts_tablet_manager.cc:1875] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:43:01.023227 441 ts_tablet_manager.cc:1888] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:43:01.023296 441 log.cc:1196] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/90da4040f73e4e549f56c1e3c121cbe9
I20240506 15:43:01.023527 441 ts_tablet_manager.cc:1909] T 90da4040f73e4e549f56c1e3c121cbe9 P 76ec08e9f561432eacc0cb9bee1164b1: Deleting consensus metadata
I20240506 15:43:01.024048 32698 catalog_manager.cc:4830] TS 76ec08e9f561432eacc0cb9bee1164b1 (127.17.240.193:37011): tablet 90da4040f73e4e549f56c1e3c121cbe9 (table test-table [id=abc175982b4a43f5b296c67e948b2c69]) successfully deleted
W20240506 15:43:01.092355 436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:01.092566 436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:01.092588 436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:01.093791 436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:01.093825 436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:01.093849 436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:43:01.095320 436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:36209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=33201
--tserver_master_addrs=127.17.240.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:01.095507 436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:01.095688 436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:01.100282 446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.100733 449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.101277 447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:01.102063 436 server_base.cc:1034] running on GCE node
I20240506 15:43:01.102222 436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:01.102434 436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:01.103581 436 hybrid_clock.cc:648] HybridClock initialized: now 1715010181103566 us; error 35 us; skew 500 ppm
May 06 15:43:01 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:01.105875 436 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:43:01.106235 436 webserver.cc:460] Webserver started at http://127.17.240.194:33201/ using document root <none> and password file <none>
I20240506 15:43:01.106390 436 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:01.106443 436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:01.107658 436 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:01.108145 456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.108341 436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:43:01.108435 436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "19c190e86a9a4160b77226a199fe726d"
format_stamp: "Formatted at 2024-05-06 15:43:00 on dist-test-slave-v0k1"
server_key: "e7ed18e278f9e729fec1432864a3753e"
server_key_iv: "292d02e8d3ad5dca1fbc91948ce1c1c3"
server_key_version: "encryptionkey@0"
I20240506 15:43:01.108721 436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:01.116283 436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:01.117913 436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:01.118026 436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:01.118201 436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:01.118471 436 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:01.118502 436 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.118534 436 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:01.118563 436 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.123914 436 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:36209
I20240506 15:43:01.123977 569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:36209 every 8 connection(s)
I20240506 15:43:01.124269 436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
May 06 15:43:01 dist-test-slave-v0k1 krb5kdc[32132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:01.133009 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.124902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:46395 (local address 127.17.240.254:38247)
0506 15:43:01.125004 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.125007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.125158 (+ 151us) server_negotiation.cc:407] Connection header received
0506 15:43:01.125660 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.125662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.125677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.125737 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:01.126763 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.127302 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.127837 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.128026 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.130049 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:01.130053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:01.130054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:01.130061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:01.131744 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:01.132181 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:01.132182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:01.132183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:01.132207 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:01.132441 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:01.132441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:01.132441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:01.132529 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:01.132584 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.132857 (+ 273us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.132932 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20240506 15:43:01.133097 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.124859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:46395)
0506 15:43:01.125049 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.125081 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:01.125557 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:01.125795 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:01.125799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:01.126081 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:01.126659 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.126664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.127390 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.127391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:01.127757 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.127758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.127885 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.128458 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:01.128488 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:01.129937 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:01.131836 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:01.131843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:01.131845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:01.132053 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:01.132288 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:01.132289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:01.132289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:01.132380 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:01.132618 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:01.132624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:01.132745 (+ 121us) client_negotiation.cc:769] Sending connection context
0506 15:43:01.132843 (+ 98us) client_negotiation.cc:240] Negotiation successful
0506 15:43:01.132952 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20240506 15:43:01.133621 570 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38247
I20240506 15:43:01.133711 570 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:01.133906 570 heartbeater.cc:511] Master 127.17.240.254:38247 requested a full tablet report, sending...
I20240506 15:43:01.134268 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 436
I20240506 15:43:01.134344 32712 ts_manager.cc:194] Registered new tserver with Master: 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
I20240506 15:43:01.134593 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:37101
--local_ip_for_outbound_sockets=127.17.240.195
--tserver_master_addrs=127.17.240.254:38247
--webserver_port=46663
--webserver_interface=127.17.240.195
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:43:01.134856 32712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:46395
W20240506 15:43:01.217303 573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:01.217499 573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:01.217520 573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:01.218844 573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:01.218883 573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:01.218910 573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:43:01.220381 573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:37101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=46663
--tserver_master_addrs=127.17.240.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:01.220580 573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:01.220765 573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:01.225419 578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.226024 581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.226569 579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:01.227327 573 server_base.cc:1034] running on GCE node
I20240506 15:43:01.227509 573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:01.227706 573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:01.228832 573 hybrid_clock.cc:648] HybridClock initialized: now 1715010181228816 us; error 42 us; skew 500 ppm
May 06 15:43:01 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:01.231181 573 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:43:01.231606 573 webserver.cc:460] Webserver started at http://127.17.240.195:46663/ using document root <none> and password file <none>
I20240506 15:43:01.231760 573 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:01.231817 573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:01.233033 573 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:01.233568 588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.233695 573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.233754 573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "4fae4ef17fbb44bf8dc41d32806c42d9"
format_stamp: "Formatted at 2024-05-06 15:43:00 on dist-test-slave-v0k1"
server_key: "176d30161cf6e15808a5b9fc6f496973"
server_key_iv: "c4e4e413b9d937d9ac727d4da605fe46"
server_key_version: "encryptionkey@0"
I20240506 15:43:01.234001 573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:01.256570 573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:01.258421 573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:01.258527 573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:01.258708 573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:01.258956 573 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:01.258987 573 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.259038 573 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:01.259073 573 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.264870 573 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:37101
I20240506 15:43:01.264922 701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:37101 every 8 connection(s)
I20240506 15:43:01.265255 573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:43:01.269584 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 573
May 06 15:43:01 dist-test-slave-v0k1 krb5kdc[32132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:01.273936 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.265888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:35793 (local address 127.17.240.254:38247)
0506 15:43:01.265952 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.265954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.266138 (+ 184us) server_negotiation.cc:407] Connection header received
0506 15:43:01.266684 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.266685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.266695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.266758 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:01.267803 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.268294 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.268853 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.268961 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.271092 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:01.271097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:01.271097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:01.271105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:01.272751 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:01.273128 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:01.273129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:01.273129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:01.273152 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:01.273337 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:01.273338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:01.273339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:01.273428 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:01.273489 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.273744 (+ 255us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.273840 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240506 15:43:01.274070 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.265886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:35793)
0506 15:43:01.266029 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.266058 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:01.266580 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:01.266806 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:01.266809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:01.267127 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:01.267699 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.267704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.268396 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.268397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:01.268779 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.268780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.268897 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.269426 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:01.269445 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:01.270978 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:01.272836 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:01.272840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:01.272843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:01.273046 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:01.273211 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:01.273211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:01.273212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:01.273278 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:01.273486 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:01.273492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:01.273616 (+ 124us) client_negotiation.cc:769] Sending connection context
0506 15:43:01.273715 (+ 99us) client_negotiation.cc:240] Negotiation successful
0506 15:43:01.273850 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20240506 15:43:01.274556 702 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38247
I20240506 15:43:01.274652 702 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:01.274845 702 heartbeater.cc:511] Master 127.17.240.254:38247 requested a full tablet report, sending...
I20240506 15:43:01.275279 32712 ts_manager.cc:194] Registered new tserver with Master: 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101)
I20240506 15:43:01.275812 32712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:35793
I20240506 15:43:01.276688 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.269966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.17.240.254:38247)
0506 15:43:01.270680 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.270682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.270691 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:43:01.270715 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.270716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.270729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.270792 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:01.271682 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.272197 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.272988 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.273180 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.274074 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:01.274079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:01.274080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:01.274085 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:01.275768 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:01.276098 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:01.276099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:01.276099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:01.276121 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:01.276284 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:01.276284 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:01.276284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:01.276352 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:01.276450 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.276561 (+ 111us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.276624 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":67,"threads_started":1}
I20240506 15:43:01.277354 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:43:01.279762 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.277695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.17.240.254:38247)
0506 15:43:01.277748 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.277749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.277863 (+ 114us) server_negotiation.cc:407] Connection header received
0506 15:43:01.277911 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.277911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.277941 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.278005 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:43:01.278281 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.278761 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.279367 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.279473 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.279506 (+ 33us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:43:01.279578 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:43:01.279643 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.279696 (+ 53us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.279715 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240506 15:43:01.280028 32712 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:43:01.281448 32712 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:43:01.283514 366 tablet_service.cc:1467] Processing CreateTablet for tablet 366101457baa466da8af0f55dcc0d180 (DEFAULT_TABLE table=test-table [id=63ce4cdd03924151a9a1bb607aa692f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:01.283704 366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366101457baa466da8af0f55dcc0d180. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:01.285205 324 tablet_bootstrap.cc:492] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: Bootstrap starting.
I20240506 15:43:01.285573 324 tablet_bootstrap.cc:654] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:01.286156 324 tablet_bootstrap.cc:492] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: No bootstrap required, opened a new log
I20240506 15:43:01.286164 708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.283118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:37101 (local address 127.0.0.1:46854)
0506 15:43:01.283297 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.283309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:01.283367 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:01.283622 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:01.283623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:01.283625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:01.283821 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.283823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.285034 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.285035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:01.285848 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.285850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.285937 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.285942 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:43:01.285975 (+ 33us) client_negotiation.cc:240] Negotiation successful
0506 15:43:01.286096 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20240506 15:43:01.286208 324 ts_tablet_manager.cc:1367] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:01.286306 324 raft_consensus.cc:348] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.286351 324 raft_consensus.cc:374] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:01.286365 324 raft_consensus.cc:729] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ec08e9f561432eacc0cb9bee1164b1, State: Initialized, Role: FOLLOWER
I20240506 15:43:01.286378 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.283140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:36209 (local address 127.0.0.1:35058)
0506 15:43:01.283343 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.283353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:01.283383 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:01.283902 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:01.283903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:01.283905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:01.284066 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.284067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.285478 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.285479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:01.286175 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.286177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.286240 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.286243 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:43:01.286275 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:43:01.286313 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":105,"threads_started":1}
I20240506 15:43:01.286412 324 consensus_queue.cc:260] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.286502 324 ts_tablet_manager.cc:1398] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.287093 710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.283236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.17.240.194:36209)
0506 15:43:01.283743 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.283752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.283761 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:43:01.283800 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.283801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.283818 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.283901 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:01.284245 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.285397 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.286275 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.286896 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.286935 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.286961 (+ 26us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.286989 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":96,"threads_started":1}
I20240506 15:43:01.287664 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.283187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.17.240.195:37101)
0506 15:43:01.283485 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.283492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.283499 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:43:01.283528 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.283529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.283543 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.283617 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:01.284173 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.284956 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.285951 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.287523 (+ 1572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.287554 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.287581 (+ 27us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.287613 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":71,"threads_started":1}
I20240506 15:43:01.287619 504 tablet_service.cc:1467] Processing CreateTablet for tablet 366101457baa466da8af0f55dcc0d180 (DEFAULT_TABLE table=test-table [id=63ce4cdd03924151a9a1bb607aa692f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:01.287884 504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366101457baa466da8af0f55dcc0d180. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:01.288167 636 tablet_service.cc:1467] Processing CreateTablet for tablet 366101457baa466da8af0f55dcc0d180 (DEFAULT_TABLE table=test-table [id=63ce4cdd03924151a9a1bb607aa692f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:01.288414 636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366101457baa466da8af0f55dcc0d180. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:01.290331 712 tablet_bootstrap.cc:492] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: Bootstrap starting.
I20240506 15:43:01.290946 713 tablet_bootstrap.cc:492] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: Bootstrap starting.
I20240506 15:43:01.290995 712 tablet_bootstrap.cc:654] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:01.291280 712 log.cc:824] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:01.291474 713 tablet_bootstrap.cc:654] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:01.291857 713 log.cc:824] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:01.292044 712 tablet_bootstrap.cc:492] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: No bootstrap required, opened a new log
I20240506 15:43:01.292117 712 ts_tablet_manager.cc:1367] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:43:01.292516 713 tablet_bootstrap.cc:492] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: No bootstrap required, opened a new log
I20240506 15:43:01.292577 713 ts_tablet_manager.cc:1367] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:43:01.293738 712 raft_consensus.cc:348] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.293853 712 raft_consensus.cc:374] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:01.293880 712 raft_consensus.cc:729] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c190e86a9a4160b77226a199fe726d, State: Initialized, Role: FOLLOWER
I20240506 15:43:01.293998 712 consensus_queue.cc:260] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.294227 712 ts_tablet_manager.cc:1398] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 15:43:01.294193 713 raft_consensus.cc:348] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.294299 570 heartbeater.cc:503] Master 127.17.240.254:38247 was elected leader, sending a full tablet report...
I20240506 15:43:01.294330 713 raft_consensus.cc:374] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:01.294363 713 raft_consensus.cc:729] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fae4ef17fbb44bf8dc41d32806c42d9, State: Initialized, Role: FOLLOWER
I20240506 15:43:01.294454 713 consensus_queue.cc:260] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.294710 713 ts_tablet_manager.cc:1398] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:43:01.294765 702 heartbeater.cc:503] Master 127.17.240.254:38247 was elected leader, sending a full tablet report...
I20240506 15:43:01.357780 440 raft_consensus.cc:482] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:43:01.357926 440 raft_consensus.cc:504] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.358240 440 leader_election.cc:288] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101), 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
I20240506 15:43:01.360631 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.358433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:36209 (local address 127.17.240.193:43391)
0506 15:43:01.358481 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.358495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:01.358545 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:01.358711 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:01.358711 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:01.358713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:01.358946 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.358948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.359905 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.359906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:01.360447 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.360448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.360519 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.360522 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:43:01.360551 (+ 29us) client_negotiation.cc:240] Negotiation successful
0506 15:43:01.360579 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240506 15:43:01.361147 710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.358516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:43391 (local address 127.17.240.194:36209)
0506 15:43:01.358571 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.358573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.358584 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:43:01.358613 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.358613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.358623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.358694 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:01.359045 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.359817 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.360551 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.361007 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.361016 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.361037 (+ 21us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.361066 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240506 15:43:01.361311 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.358437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:37101 (local address 127.17.240.193:42613)
0506 15:43:01.358640 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.358650 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:01.358713 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:01.358876 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:01.358877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:01.358879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:01.359087 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.359089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.359736 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.359738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:01.361060 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:01.361063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.361152 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.361155 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:43:01.361191 (+ 36us) client_negotiation.cc:240] Negotiation successful
0506 15:43:01.361230 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":15,"thread_start_us":111,"threads_started":1}
I20240506 15:43:01.361442 524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366101457baa466da8af0f55dcc0d180" candidate_uuid: "76ec08e9f561432eacc0cb9bee1164b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c190e86a9a4160b77226a199fe726d" is_pre_election: true
I20240506 15:43:01.361583 524 raft_consensus.cc:2454] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ec08e9f561432eacc0cb9bee1164b1 in term 0.
I20240506 15:43:01.361783 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.358577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:42613 (local address 127.17.240.195:37101)
0506 15:43:01.358619 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.358621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.358717 (+ 96us) server_negotiation.cc:407] Connection header received
0506 15:43:01.358771 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.358771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.358781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.358836 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:01.359166 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.359645 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.361153 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.361653 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.361661 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.361686 (+ 25us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.361720 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:43:01.361804 320 leader_election.cc:302] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19c190e86a9a4160b77226a199fe726d, 76ec08e9f561432eacc0cb9bee1164b1; no voters:
I20240506 15:43:01.361934 440 raft_consensus.cc:2781] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:43:01.361995 440 raft_consensus.cc:482] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:43:01.362020 440 raft_consensus.cc:3037] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:01.362071 649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366101457baa466da8af0f55dcc0d180" candidate_uuid: "76ec08e9f561432eacc0cb9bee1164b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" is_pre_election: true
I20240506 15:43:01.362167 649 raft_consensus.cc:2454] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ec08e9f561432eacc0cb9bee1164b1 in term 0.
I20240506 15:43:01.362751 440 raft_consensus.cc:504] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.362879 440 leader_election.cc:288] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 1 election: Requested vote from peers 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101), 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
I20240506 15:43:01.363086 649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366101457baa466da8af0f55dcc0d180" candidate_uuid: "76ec08e9f561432eacc0cb9bee1164b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9"
I20240506 15:43:01.363121 524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366101457baa466da8af0f55dcc0d180" candidate_uuid: "76ec08e9f561432eacc0cb9bee1164b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c190e86a9a4160b77226a199fe726d"
I20240506 15:43:01.363178 649 raft_consensus.cc:3037] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:01.363196 524 raft_consensus.cc:3037] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:01.364064 524 raft_consensus.cc:2454] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ec08e9f561432eacc0cb9bee1164b1 in term 1.
I20240506 15:43:01.364064 649 raft_consensus.cc:2454] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ec08e9f561432eacc0cb9bee1164b1 in term 1.
I20240506 15:43:01.364271 319 leader_election.cc:302] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fae4ef17fbb44bf8dc41d32806c42d9, 76ec08e9f561432eacc0cb9bee1164b1; no voters:
I20240506 15:43:01.364416 440 raft_consensus.cc:2781] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:43:01.364650 440 raft_consensus.cc:686] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 LEADER]: Becoming Leader. State: Replica: 76ec08e9f561432eacc0cb9bee1164b1, State: Running, Role: LEADER
I20240506 15:43:01.364742 440 consensus_queue.cc:237] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.365581 32712 catalog_manager.cc:5484] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76ec08e9f561432eacc0cb9bee1164b1 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "76ec08e9f561432eacc0cb9bee1164b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } health_report { overall_health: HEALTHY } } }
W20240506 15:43:01.374833 571 tablet.cc:2369] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:43:01.379477 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.377520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.17.240.193:37011)
0506 15:43:01.377579 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:01.377581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:01.377589 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:43:01.377634 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:01.377635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:01.377652 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:01.377717 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:43:01.377943 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.378433 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:01.378996 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:01.379155 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:01.379194 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:43:01.379280 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:43:01.379338 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:01.379390 (+ 52us) server_negotiation.cc:299] Negotiation successful
0506 15:43:01.379410 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:43:01.380980 649 raft_consensus.cc:1261] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 1 FOLLOWER]: Refusing update from remote peer 76ec08e9f561432eacc0cb9bee1164b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:43:01.380981 524 raft_consensus.cc:1261] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d [term 1 FOLLOWER]: Refusing update from remote peer 76ec08e9f561432eacc0cb9bee1164b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:43:01.381300 719 consensus_queue.cc:1035] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:43:01.381433 440 consensus_queue.cc:1035] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:43:01.384158 723 mvcc.cc:205] Tried to move back new op lower bound from 7024681702934376448 to 7024681702870675456. Current Snapshot: MvccSnapshot[applied={T|T < 7024681702934376448}]
I20240506 15:43:01.385967 32711 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40494:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:43:01.386068 32711 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40494:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:43:01.386837 32711 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Sending DeleteTablet for 3 replicas of tablet 366101457baa466da8af0f55dcc0d180
I20240506 15:43:01.387156 504 tablet_service.cc:1514] Processing DeleteTablet for tablet 366101457baa466da8af0f55dcc0d180 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:01 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:35058
I20240506 15:43:01.387190 366 tablet_service.cc:1514] Processing DeleteTablet for tablet 366101457baa466da8af0f55dcc0d180 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:01 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:41248
I20240506 15:43:01.387208 636 tablet_service.cc:1514] Processing DeleteTablet for tablet 366101457baa466da8af0f55dcc0d180 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:01 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:46854
I20240506 15:43:01.387287 441 tablet_replica.cc:330] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: stopping tablet replica
I20240506 15:43:01.387338 441 raft_consensus.cc:2229] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:43:01.387414 441 raft_consensus.cc:2258] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:01.387426 732 tablet_replica.cc:330] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: stopping tablet replica
I20240506 15:43:01.387445 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 32750
I20240506 15:43:01.387506 732 raft_consensus.cc:2229] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:43:01.387593 732 raft_consensus.cc:2258] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:01.387722 733 tablet_replica.cc:330] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: stopping tablet replica
I20240506 15:43:01.387794 733 raft_consensus.cc:2229] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:43:01.387863 733 raft_consensus.cc:2258] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:01.388835 732 ts_tablet_manager.cc:1875] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:43:01.390180 733 ts_tablet_manager.cc:1875] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:43:01.390782 732 ts_tablet_manager.cc:1888] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:43:01.390858 732 log.cc:1196] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/366101457baa466da8af0f55dcc0d180
I20240506 15:43:01.391196 732 ts_tablet_manager.cc:1909] T 366101457baa466da8af0f55dcc0d180 P 19c190e86a9a4160b77226a199fe726d: Deleting consensus metadata
I20240506 15:43:01.391670 32698 catalog_manager.cc:4830] TS 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209): tablet 366101457baa466da8af0f55dcc0d180 (table test-table [id=63ce4cdd03924151a9a1bb607aa692f6]) successfully deleted
I20240506 15:43:01.391968 733 ts_tablet_manager.cc:1888] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:43:01.392035 733 log.cc:1196] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/366101457baa466da8af0f55dcc0d180
I20240506 15:43:01.392256 733 ts_tablet_manager.cc:1909] T 366101457baa466da8af0f55dcc0d180 P 4fae4ef17fbb44bf8dc41d32806c42d9: Deleting consensus metadata
I20240506 15:43:01.392882 32697 catalog_manager.cc:4830] TS 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101): tablet 366101457baa466da8af0f55dcc0d180 (table test-table [id=63ce4cdd03924151a9a1bb607aa692f6]) successfully deleted
W20240506 15:43:01.393028 32698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:37011 (error 108)
I20240506 15:43:01.393263 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 436
I20240506 15:43:01.394002 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.393445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37011 (local address 127.0.0.1:41270)
0506 15:43:01.393914 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.393959 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:37011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426}
W20240506 15:43:01.394168 32698 catalog_manager.cc:4573] TS 76ec08e9f561432eacc0cb9bee1164b1 (127.17.240.193:37011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 366101457baa466da8af0f55dcc0d180: Network error: Client connection negotiation failed: client connection to 127.17.240.193:37011: connect: Connection refused (error 111)
I20240506 15:43:01.398779 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 573
I20240506 15:43:01.403756 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 32679
I20240506 15:43:01.409494 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38247
--webserver_interface=127.17.240.254
--webserver_port=39109
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:38247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:01.490290 735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:01.490479 735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:01.490501 735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:01.491859 735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:43:01.491910 735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:01.491925 735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:01.491936 735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:43:01.491947 735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:43:01.493433 735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:38247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=39109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:01.493641 735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:01.493825 735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:01.498512 740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.499317 743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.499827 741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:01.500425 735 server_base.cc:1034] running on GCE node
I20240506 15:43:01.500679 735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:01.500917 735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:01.502044 735 hybrid_clock.cc:648] HybridClock initialized: now 1715010181502033 us; error 35 us; skew 500 ppm
May 06 15:43:01 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:01.504477 735 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:43:01.504911 735 webserver.cc:460] Webserver started at http://127.17.240.254:39109/ using document root <none> and password file <none>
I20240506 15:43:01.505100 735 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:01.505146 735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:01.506412 735 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:01.507099 750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.507262 735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:01.507340 735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "89668ff351f7428fae5622bbc4b3a14c"
format_stamp: "Formatted at 2024-05-06 15:42:59 on dist-test-slave-v0k1"
server_key: "2e0236d2994eb51a82e869db561a8569"
server_key_iv: "ebcc61d358f0c2bd09dfe5c093e97e8a"
server_key_version: "encryptionkey@0"
I20240506 15:43:01.507639 735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:01.522491 735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:01.524523 735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:01.524631 735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:01.528443 735 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:38247
I20240506 15:43:01.528510 802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:38247 every 8 connection(s)
I20240506 15:43:01.528818 735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:43:01.530233 803 sys_catalog.cc:263] Verifying existing consensus state
I20240506 15:43:01.530784 803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Bootstrap starting.
I20240506 15:43:01.534372 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 735
I20240506 15:43:01.534732 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:37011
--local_ip_for_outbound_sockets=127.17.240.193
--tserver_master_addrs=127.17.240.254:38247
--webserver_port=40017
--webserver_interface=127.17.240.193
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:43:01.540820 803 log.cc:824] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:01.542845 803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:43:01.543128 803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c: Bootstrap complete.
I20240506 15:43:01.545222 803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:43:01.545423 803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89668ff351f7428fae5622bbc4b3a14c, State: Initialized, Role: FOLLOWER
I20240506 15:43:01.545552 803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:43:01.545622 803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:43:01.545666 803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:43:01.545714 803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:43:01.546586 803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:43:01.546677 803 leader_election.cc:302] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89668ff351f7428fae5622bbc4b3a14c; no voters:
I20240506 15:43:01.546821 803 leader_election.cc:288] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [CANDIDATE]: Term 3 election: Requested vote from peers
I20240506 15:43:01.546876 807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:43:01.547067 807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [term 3 LEADER]: Becoming Leader. State: Replica: 89668ff351f7428fae5622bbc4b3a14c, State: Running, Role: LEADER
I20240506 15:43:01.547083 803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:43:01.547222 807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } }
I20240506 15:43:01.547807 808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "89668ff351f7428fae5622bbc4b3a14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } } }
I20240506 15:43:01.547818 809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89668ff351f7428fae5622bbc4b3a14c. Latest consensus state: current_term: 3 leader_uuid: "89668ff351f7428fae5622bbc4b3a14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89668ff351f7428fae5622bbc4b3a14c" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38247 } } }
I20240506 15:43:01.547986 809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: This master's current role is: LEADER
I20240506 15:43:01.547977 808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89668ff351f7428fae5622bbc4b3a14c [sys.catalog]: This master's current role is: LEADER
I20240506 15:43:01.548187 817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:43:01.548897 817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:43:01.549000 817 catalog_manager.cc:1259] Loaded cluster ID: 898d06095a284549bb45413f1912dbc4
I20240506 15:43:01.549065 817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20240506 15:43:01.549302 817 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6b7d53fdcd google::LogMessage::Fail() at ??:0
@ 0x7f6b7d543b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6b7d53f7cc google::LogMessage::Flush() at ??:0
@ 0x7f6b7d540f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6b7fd18af6 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6b7fd23294 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6b7e639a08 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6b7e62e3a4 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6b7e9516db start_thread at ??:0
@ 0x7f6b7ca9471f clone at ??:0
W20240506 15:43:01.623670 805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:01.623867 805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:01.623899 805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:01.625186 805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:01.625229 805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:01.625267 805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:43:01.626719 805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:37011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=40017
--tserver_master_addrs=127.17.240.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:01.626948 805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:01.627147 805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:01.632400 842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.633042 839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.633792 840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:01.634335 805 server_base.cc:1034] running on GCE node
I20240506 15:43:01.634541 805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:01.634769 805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:01.635919 805 hybrid_clock.cc:648] HybridClock initialized: now 1715010181635903 us; error 35 us; skew 500 ppm
May 06 15:43:01 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:01.638306 805 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:43:01.638724 805 webserver.cc:460] Webserver started at http://127.17.240.193:40017/ using document root <none> and password file <none>
I20240506 15:43:01.638960 805 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:01.639032 805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:01.640417 805 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:01.641065 849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.641243 805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:43:01.641352 805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "76ec08e9f561432eacc0cb9bee1164b1"
format_stamp: "Formatted at 2024-05-06 15:42:59 on dist-test-slave-v0k1"
server_key: "c4ed6f8d453046891f6c6d73b1ff4371"
server_key_iv: "a00ee4a6705b7f874fbd72e862998c75"
server_key_version: "encryptionkey@0"
I20240506 15:43:01.641651 805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:01.679510 805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:01.681663 805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:01.681792 805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:01.682037 805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:01.682440 856 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240506 15:43:01.683626 805 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 15:43:01.683686 805 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:01.683727 805 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 15:43:01.684394 805 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 15:43:01.684481 856 tablet_bootstrap.cc:492] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: Bootstrap starting.
I20240506 15:43:01.684509 805 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:01.692068 805 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:37011
I20240506 15:43:01.692147 963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:37011 every 8 connection(s)
I20240506 15:43:01.692466 805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:43:01.695863 856 log.cc:824] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:01.698009 856 tablet_bootstrap.cc:492] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:43:01.698276 856 tablet_bootstrap.cc:492] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: Bootstrap complete.
I20240506 15:43:01.699237 856 ts_tablet_manager.cc:1367] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.012s
I20240506 15:43:01.700109 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 805
I20240506 15:43:01.700410 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:36209
--local_ip_for_outbound_sockets=127.17.240.194
--tserver_master_addrs=127.17.240.254:38247
--webserver_port=33201
--webserver_interface=127.17.240.194
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:43:01.700358 856 raft_consensus.cc:348] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.700688 856 raft_consensus.cc:729] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ec08e9f561432eacc0cb9bee1164b1, State: Initialized, Role: FOLLOWER
I20240506 15:43:01.700824 856 consensus_queue.cc:260] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:01.701086 856 ts_tablet_manager.cc:1398] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20240506 15:43:01.754155 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:43:01.695678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:40031)
0506 15:43:01.695892 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.695925 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:01.696483 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:01.753994 (+ 57511us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
W20240506 15:43:01.754469 964 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:38247 (0 consecutive failures): Network error: Failed to ping master at 127.17.240.254:38247: Client connection negotiation failed: client connection to 127.17.240.254:38247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 15:43:01.754860 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.754731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:46045)
0506 15:43:01.754791 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.754834 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20240506 15:43:01.755210 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.755134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:59815)
0506 15:43:01.755171 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.755193 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20240506 15:43:01.755317 964 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240506 15:43:01.781649 969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:01.781842 969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:01.781872 969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:01.783198 969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:01.783243 969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:01.783273 969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:43:01.784751 969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:36209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=33201
--tserver_master_addrs=127.17.240.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:01.784948 969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:01.785143 969 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:01.790333 974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.790833 975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.791612 977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:01.792320 969 server_base.cc:1034] running on GCE node
I20240506 15:43:01.792517 969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:01.792717 969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:01.793860 969 hybrid_clock.cc:648] HybridClock initialized: now 1715010181793840 us; error 36 us; skew 500 ppm
May 06 15:43:01 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:01.796515 969 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:43:01.796931 969 webserver.cc:460] Webserver started at http://127.17.240.194:33201/ using document root <none> and password file <none>
I20240506 15:43:01.797112 969 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:01.797185 969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:01.798457 969 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:01.799058 984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.799209 969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:43:01.799283 969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "19c190e86a9a4160b77226a199fe726d"
format_stamp: "Formatted at 2024-05-06 15:43:00 on dist-test-slave-v0k1"
server_key: "e7ed18e278f9e729fec1432864a3753e"
server_key_iv: "292d02e8d3ad5dca1fbc91948ce1c1c3"
server_key_version: "encryptionkey@0"
I20240506 15:43:01.799623 969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:01.817898 969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:01.819844 969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:01.820039 969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:01.820276 969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:01.820613 969 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:01.820647 969 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.820674 969 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:01.820704 969 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.826562 969 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:36209
I20240506 15:43:01.826632 1097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:36209 every 8 connection(s)
I20240506 15:43:01.826959 969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:43:01.828075 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.827703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:44187)
0506 15:43:01.827889 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.827952 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
W20240506 15:43:01.828308 1098 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:38247 (0 consecutive failures): Network error: Failed to ping master at 127.17.240.254:38247: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
I20240506 15:43:01.828598 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.828502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:58745)
0506 15:43:01.828552 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.828577 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20240506 15:43:01.828941 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.828825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:48559)
0506 15:43:01.828887 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.828918 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20240506 15:43:01.829057 1098 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240506 15:43:01.835727 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 969
I20240506 15:43:01.836001 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:37101
--local_ip_for_outbound_sockets=127.17.240.195
--tserver_master_addrs=127.17.240.254:38247
--webserver_port=46663
--webserver_interface=127.17.240.195
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:01.914753 1101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:01.914939 1101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:01.914963 1101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:01.916220 1101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:01.916263 1101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:01.916287 1101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:43:01.917662 1101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:37101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=46663
--tserver_master_addrs=127.17.240.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:01.917824 1101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:01.917965 1101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:01.922868 1106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.923403 1107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:01.923998 1109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:01.924628 1101 server_base.cc:1034] running on GCE node
I20240506 15:43:01.924825 1101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:01.925032 1101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:01.926177 1101 hybrid_clock.cc:648] HybridClock initialized: now 1715010181926160 us; error 34 us; skew 500 ppm
May 06 15:43:01 dist-test-slave-v0k1 krb5kdc[32132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:01.928576 1101 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:43:01.928956 1101 webserver.cc:460] Webserver started at http://127.17.240.195:46663/ using document root <none> and password file <none>
I20240506 15:43:01.929131 1101 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:01.929174 1101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:01.930500 1101 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:01.931061 1116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.931210 1101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.931291 1101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "4fae4ef17fbb44bf8dc41d32806c42d9"
format_stamp: "Formatted at 2024-05-06 15:43:00 on dist-test-slave-v0k1"
server_key: "176d30161cf6e15808a5b9fc6f496973"
server_key_iv: "c4e4e413b9d937d9ac727d4da605fe46"
server_key_version: "encryptionkey@0"
I20240506 15:43:01.931587 1101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:01.941411 1101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:01.943403 1101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:01.943517 1101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:01.943737 1101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:01.944031 1101 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:01.944065 1101 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.944098 1101 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:01.944133 1101 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:01.949333 1101 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:37101
I20240506 15:43:01.949373 1229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:37101 every 8 connection(s)
I20240506 15:43:01.949688 1101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:43:01.950552 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 1101
I20240506 15:43:01.950569 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.950283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:60045)
0506 15:43:01.950401 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.950458 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
W20240506 15:43:01.950773 1230 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:38247 (0 consecutive failures): Network error: Failed to ping master at 127.17.240.254:38247: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
I20240506 15:43:01.951131 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.950983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:44695)
0506 15:43:01.951067 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.951101 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20240506 15:43:01.951395 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:01.951328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:58033)
0506 15:43:01.951363 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:01.951382 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20240506 15:43:01.951484 1230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240506 15:43:02.755887 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:02.755658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:48323)
0506 15:43:02.755784 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:02.755834 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20240506 15:43:02.829651 1235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:02.829404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:39737)
0506 15:43:02.829546 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:02.829577 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20240506 15:43:02.936973 1236 raft_consensus.cc:482] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:43:02.937095 1236 raft_consensus.cc:504] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:02.937441 1236 leader_election.cc:288] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101), 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
May 06 15:43:02 dist-test-slave-v0k1 krb5kdc[32132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.194@KRBTEST.COM
May 06 15:43:02 dist-test-slave-v0k1 krb5kdc[32132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.195@KRBTEST.COM
I20240506 15:43:02.946606 1237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:02.937602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:36209 (local address 127.17.240.193:54147)
0506 15:43:02.937797 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:02.937807 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:02.937871 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:02.938496 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:02.938497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:02.938925 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:02.939211 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:02.939217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:02.940182 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:02.940183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:02.940515 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:02.940516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:02.940630 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:02.941329 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:02.941356 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:02.943056 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:02.945129 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:02.945136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:02.945139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:02.945372 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:02.945584 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:02.945585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:02.945585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:02.945678 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:02.946082 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:02.946088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:02.946214 (+ 126us) client_negotiation.cc:769] Sending connection context
0506 15:43:02.946326 (+ 112us) client_negotiation.cc:240] Negotiation successful
0506 15:43:02.946500 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":11,"thread_start_us":68,"threads_started":1}
I20240506 15:43:02.946630 1238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:02.937706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:33549 (local address 127.17.240.195:37101)
0506 15:43:02.937836 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:02.937846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:02.937865 (+ 19us) server_negotiation.cc:407] Connection header received
0506 15:43:02.938265 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:02.938267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:02.938312 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:02.938446 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:02.939322 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:02.940272 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:02.940782 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:02.941037 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:02.943502 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:02.943530 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:02.943531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:02.943548 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:02.945354 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:02.945697 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:02.945700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:02.945700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:02.945762 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:02.945901 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:02.945902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:02.945903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:02.946116 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:02.946207 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:02.946381 (+ 174us) server_negotiation.cc:299] Negotiation successful
0506 15:43:02.946500 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":43,"threads_started":1}
I20240506 15:43:02.946636 1239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:02.937719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:54147 (local address 127.17.240.194:36209)
0506 15:43:02.937840 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:02.937868 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:02.937883 (+ 15us) server_negotiation.cc:407] Connection header received
0506 15:43:02.938286 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:02.938288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:02.938319 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:02.938449 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:02.939353 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:02.940078 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:02.940587 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:02.940823 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:02.943184 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:02.943193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:02.943193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:02.943210 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:02.945053 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:02.945452 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:02.945455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:02.945455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:02.945511 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:02.945749 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:02.945751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:02.945751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:02.946014 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:02.946088 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:02.946391 (+ 303us) server_negotiation.cc:299] Negotiation successful
0506 15:43:02.946496 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
I20240506 15:43:02.946730 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:02.937599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:37101 (local address 127.17.240.193:33549)
0506 15:43:02.937666 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:02.937677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:02.937735 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:02.938491 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:02.938494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:02.938924 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:02.939211 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:02.939217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:02.940372 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:02.940373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:02.940710 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:02.940711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:02.940775 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:02.941610 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:02.941614 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:02.943381 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:02.945437 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:02.945438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:02.945438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:02.945605 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:02.945822 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:02.945822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:02.945823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:02.945849 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:02.946194 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:02.946195 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:02.946233 (+ 38us) client_negotiation.cc:769] Sending connection context
0506 15:43:02.946333 (+ 100us) client_negotiation.cc:240] Negotiation successful
0506 15:43:02.946579 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":129}
W20240506 15:43:02.947939 852 leader_election.cc:341] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209): Not found: Tablet not found: 366101457baa466da8af0f55dcc0d180
W20240506 15:43:02.948139 851 leader_election.cc:341] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101): Not found: Tablet not found: 366101457baa466da8af0f55dcc0d180
I20240506 15:43:02.948201 851 leader_election.cc:302] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76ec08e9f561432eacc0cb9bee1164b1; no voters: 19c190e86a9a4160b77226a199fe726d, 4fae4ef17fbb44bf8dc41d32806c42d9
I20240506 15:43:02.948307 1236 raft_consensus.cc:2726] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:43:02.952039 1240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:02.951802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:49191)
0506 15:43:02.951940 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:02.951997 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20240506 15:43:03.756695 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:03.756453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:52187)
0506 15:43:03.756600 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:03.756645 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20240506 15:43:03.830335 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:03.830105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:59197)
0506 15:43:03.830241 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:03.830290 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20240506 15:43:03.952735 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:03.952517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:47327)
0506 15:43:03.952642 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:03.952678 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20240506 15:43:04.757493 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:04.757261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:38743)
0506 15:43:04.757402 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:04.757442 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20240506 15:43:04.776976 1245 raft_consensus.cc:482] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:43:04.777077 1245 raft_consensus.cc:504] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:04.777254 1245 leader_election.cc:288] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101), 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
W20240506 15:43:04.777676 852 leader_election.cc:341] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209): Not found: Tablet not found: 366101457baa466da8af0f55dcc0d180
W20240506 15:43:04.777745 851 leader_election.cc:341] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101): Not found: Tablet not found: 366101457baa466da8af0f55dcc0d180
I20240506 15:43:04.777822 851 leader_election.cc:302] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76ec08e9f561432eacc0cb9bee1164b1; no voters: 19c190e86a9a4160b77226a199fe726d, 4fae4ef17fbb44bf8dc41d32806c42d9
I20240506 15:43:04.777922 1245 raft_consensus.cc:2726] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:43:04.831169 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:04.830871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:52929)
0506 15:43:04.831064 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:04.831103 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20240506 15:43:04.953389 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:04.953190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:45717)
0506 15:43:04.953304 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:04.953343 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20240506 15:43:05.758301 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:05.758016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:59823)
0506 15:43:05.758201 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:05.758248 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20240506 15:43:05.831848 1249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:05.831624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:39025)
0506 15:43:05.831760 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:05.831798 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20240506 15:43:05.954136 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:05.953917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:35815)
0506 15:43:05.954042 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:05.954089 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20240506 15:43:06.666927 1251 raft_consensus.cc:482] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:43:06.667057 1251 raft_consensus.cc:504] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:06.667243 1251 leader_election.cc:288] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101), 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
W20240506 15:43:06.667711 852 leader_election.cc:341] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209): Not found: Tablet not found: 366101457baa466da8af0f55dcc0d180
W20240506 15:43:06.667804 851 leader_election.cc:341] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101): Not found: Tablet not found: 366101457baa466da8af0f55dcc0d180
I20240506 15:43:06.667822 851 leader_election.cc:302] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76ec08e9f561432eacc0cb9bee1164b1; no voters: 19c190e86a9a4160b77226a199fe726d, 4fae4ef17fbb44bf8dc41d32806c42d9
I20240506 15:43:06.667948 1251 raft_consensus.cc:2726] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:43:06.759168 1252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:06.758905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:44717)
0506 15:43:06.759076 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:06.759118 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20240506 15:43:06.832602 1253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:06.832345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:58991)
0506 15:43:06.832495 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:06.832536 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20240506 15:43:06.954875 1254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:06.954628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:58361)
0506 15:43:06.954780 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:06.954825 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20240506 15:43:07.760027 1255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:07.759730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:39569)
0506 15:43:07.759927 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:07.759973 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
I20240506 15:43:07.833357 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:07.833100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:51887)
0506 15:43:07.833240 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:07.833278 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20240506 15:43:07.955663 1257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:07.955417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:33473)
0506 15:43:07.955556 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:07.955611 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20240506 15:43:08.760771 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:08.760545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:48241)
0506 15:43:08.760692 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:08.760722 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20240506 15:43:08.834151 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:08.833879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:45527)
0506 15:43:08.834057 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:08.834096 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20240506 15:43:08.956476 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:08.956213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:57949)
0506 15:43:08.956381 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:08.956426 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20240506 15:43:09.761598 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:09.761352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:53799)
0506 15:43:09.761503 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:09.761547 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20240506 15:43:09.834983 1262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:09.834747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:39299)
0506 15:43:09.834881 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:09.834924 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20240506 15:43:09.957227 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:09.956973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:55991)
0506 15:43:09.957141 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:09.957178 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20240506 15:43:10.762439 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:10.762185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:36535)
0506 15:43:10.762342 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:10.762388 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20240506 15:43:10.835847 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:10.835584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:57487)
0506 15:43:10.835742 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:10.835793 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20240506 15:43:10.957970 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:10.957707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:45547)
0506 15:43:10.957870 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:10.957914 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20240506 15:43:11.346479 1267 raft_consensus.cc:482] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:43:11.346580 1267 raft_consensus.cc:504] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:11.346716 1267 leader_election.cc:288] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101), 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
W20240506 15:43:11.347116 852 leader_election.cc:341] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209): Not found: Tablet not found: 366101457baa466da8af0f55dcc0d180
W20240506 15:43:11.347190 851 leader_election.cc:341] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101): Not found: Tablet not found: 366101457baa466da8af0f55dcc0d180
I20240506 15:43:11.347240 851 leader_election.cc:302] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76ec08e9f561432eacc0cb9bee1164b1; no voters: 19c190e86a9a4160b77226a199fe726d, 4fae4ef17fbb44bf8dc41d32806c42d9
I20240506 15:43:11.347313 1267 raft_consensus.cc:2726] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:43:11.763298 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:11.763082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:46145)
0506 15:43:11.763204 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:11.763247 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20240506 15:43:11.836676 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:11.836411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:43205)
0506 15:43:11.836577 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:11.836621 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20240506 15:43:11.958669 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:11.958462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:58525)
0506 15:43:11.958583 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:11.958618 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20240506 15:43:12.764032 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:12.763811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:35207)
0506 15:43:12.763943 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:12.763983 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20240506 15:43:12.837388 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:12.837112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:46119)
0506 15:43:12.837272 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:12.837326 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20240506 15:43:12.959528 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:12.959230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:54351)
0506 15:43:12.959432 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:12.959477 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":115,"threads_started":1}
I20240506 15:43:13.624204 1274 raft_consensus.cc:482] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:43:13.624315 1274 raft_consensus.cc:504] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fae4ef17fbb44bf8dc41d32806c42d9" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 37101 } } peers { permanent_uuid: "19c190e86a9a4160b77226a199fe726d" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 36209 } } peers { permanent_uuid: "76ec08e9f561432eacc0cb9bee1164b1" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37011 } }
I20240506 15:43:13.624471 1274 leader_election.cc:288] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101), 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209)
W20240506 15:43:13.624926 852 leader_election.cc:341] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19c190e86a9a4160b77226a199fe726d (127.17.240.194:36209): Not found: Tablet not found: 366101457baa466da8af0f55dcc0d180
W20240506 15:43:13.625007 851 leader_election.cc:341] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4fae4ef17fbb44bf8dc41d32806c42d9 (127.17.240.195:37101): Not found: Tablet not found: 366101457baa466da8af0f55dcc0d180
I20240506 15:43:13.625063 851 leader_election.cc:302] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76ec08e9f561432eacc0cb9bee1164b1; no voters: 19c190e86a9a4160b77226a199fe726d, 4fae4ef17fbb44bf8dc41d32806c42d9
I20240506 15:43:13.625135 1274 raft_consensus.cc:2726] T 366101457baa466da8af0f55dcc0d180 P 76ec08e9f561432eacc0cb9bee1164b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:43:13.764827 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:13.764578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:43301)
0506 15:43:13.764727 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:13.764774 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20240506 15:43:13.838167 1276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:13.837923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:37675)
0506 15:43:13.838068 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:13.838111 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20240506 15:43:13.960278 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:13.960057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:60761)
0506 15:43:13.960189 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:13.960228 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20240506 15:43:14.765653 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:14.765366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:37549)
0506 15:43:14.765557 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:14.765601 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20240506 15:43:14.838961 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:14.838713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:50629)
0506 15:43:14.838862 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:14.838909 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20240506 15:43:14.960953 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:14.960738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:33189)
0506 15:43:14.960868 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:14.960898 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20240506 15:43:15.766567 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:15.766235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:53615)
0506 15:43:15.766449 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:15.766501 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
I20240506 15:43:15.839720 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:15.839463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:47303)
0506 15:43:15.839600 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:15.839649 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20240506 15:43:15.961694 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:15.961424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:49647)
0506 15:43:15.961599 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:15.961642 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20240506 15:43:16.767319 1284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:16.767084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.193:45711)
0506 15:43:16.767235 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:16.767270 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20240506 15:43:16.840667 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:16.840428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.194:41759)
0506 15:43:16.840573 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:16.840605 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20240506 15:43:16.956643 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 805
I20240506 15:43:16.962131 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 969
I20240506 15:43:16.963208 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:16.962239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38247 (local address 127.17.240.195:37759)
0506 15:43:16.963104 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:16.963143 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":61,"threads_started":1}
I20240506 15:43:16.966776 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 1101
2024-05-06T15:43:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18490 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:43:17 dist-test-slave-v0k1 krb5kdc[1289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:43:17 dist-test-slave-v0k1 krb5kdc[1289](info): set up 2 sockets
May 06 15:43:17 dist-test-slave-v0k1 krb5kdc[1289](info): commencing operation
krb5kdc: starting...
W20240506 15:43:17.931386 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:43:18 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:43:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:43:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:43:18.049904 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38131
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:38131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:18.129124 1302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:18.129324 1302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:18.129361 1302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:18.130707 1302 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240506 15:43:18.130750 1302 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240506 15:43:18.130770 1302 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240506 15:43:18.130787 1302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:43:18.130808 1302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:18.130831 1302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:18.130867 1302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:43:18.130888 1302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:43:18.132578 1302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:38131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:18.132822 1302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:18.133021 1302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:18.138084 1308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:18.138676 1307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:18.139273 1302 server_base.cc:1034] running on GCE node
W20240506 15:43:18.139986 1310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:18.140300 1302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:18.140586 1302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:18.141732 1302 hybrid_clock.cc:648] HybridClock initialized: now 1715010198141699 us; error 49 us; skew 500 ppm
May 06 15:43:18 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:18.144248 1302 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:43:18.144892 1302 webserver.cc:460] Webserver started at http://127.17.240.254:39677/ using document root <none> and password file <none>
I20240506 15:43:18.145155 1302 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:18.145224 1302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:18.145356 1302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:43:18.146389 1302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "1471f9f6ca1a49c08ffe1f73833675b9"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "c08e45a5febaa03f60f047d89657c67d"
server_key_iv: "867721989ed59a73de7253f705ac200d"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.146703 1302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "1471f9f6ca1a49c08ffe1f73833675b9"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "c08e45a5febaa03f60f047d89657c67d"
server_key_iv: "867721989ed59a73de7253f705ac200d"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.147949 1302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:43:18.148635 1317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:18.148792 1302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:43:18.148869 1302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "1471f9f6ca1a49c08ffe1f73833675b9"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "c08e45a5febaa03f60f047d89657c67d"
server_key_iv: "867721989ed59a73de7253f705ac200d"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.148931 1302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:18.169656 1302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:18.171860 1302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:18.171993 1302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:18.175844 1302 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:38131
I20240506 15:43:18.175901 1369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:38131 every 8 connection(s)
I20240506 15:43:18.176208 1302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:43:18.177069 1370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:18.179304 1370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Bootstrap starting.
I20240506 15:43:18.179894 1370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:18.180168 1370 log.cc:824] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:18.180867 1370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: No bootstrap required, opened a new log
I20240506 15:43:18.182368 1370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:18.182490 1370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:18.182508 1370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1471f9f6ca1a49c08ffe1f73833675b9, State: Initialized, Role: FOLLOWER
I20240506 15:43:18.182664 1370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:18.182723 1370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:43:18.182745 1370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:43:18.182790 1370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:18.183352 1370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:18.183451 1370 leader_election.cc:302] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1471f9f6ca1a49c08ffe1f73833675b9; no voters:
I20240506 15:43:18.183601 1370 leader_election.cc:288] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:43:18.183655 1373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:43:18.183844 1370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:43:18.184118 1373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1471f9f6ca1a49c08ffe1f73833675b9, State: Running, Role: LEADER
I20240506 15:43:18.184206 1373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:18.184536 1375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1471f9f6ca1a49c08ffe1f73833675b9. Latest consensus state: current_term: 1 leader_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } } }
I20240506 15:43:18.184561 1374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } } }
I20240506 15:43:18.184671 1375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: This master's current role is: LEADER
I20240506 15:43:18.184677 1374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: This master's current role is: LEADER
I20240506 15:43:18.184849 1382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:43:18.184983 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 1302
I20240506 15:43:18.185108 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:43:18.185292 18371 external_mini_cluster.cc:1428] Setting key eaa46f8fd4908a154ada6df2bc7dec57
I20240506 15:43:18.185312 1382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:43:18.186991 1382 catalog_manager.cc:1347] Generated new cluster ID: fb592c99d5f2449fbc1f3c0a09779253
I20240506 15:43:18.187078 1382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:43:18 dist-test-slave-v0k1 krb5kdc[1289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:18.196681 1382 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:43:18.196955 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.185956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.17.240.254:38131)
0506 15:43:18.186377 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.186389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.186411 (+ 22us) server_negotiation.cc:407] Connection header received
0506 15:43:18.186512 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.186516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.186553 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.186689 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:18.187665 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.188825 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.189675 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.189913 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.192425 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:18.192432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:18.192432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:18.192453 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:18.195255 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:18.195630 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:18.195633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:18.195633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:18.195696 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:18.195855 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:18.195856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:18.195856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:18.196098 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:18.196238 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.196599 (+ 361us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.196734 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":66,"threads_started":1}
I20240506 15:43:18.197396 1382 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:43:18.202663 1382 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Generated new TSK 0
I20240506 15:43:18.202845 1382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:43:18.231137 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38131
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:18.311918 1396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:18.312135 1396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:18.312173 1396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:18.313536 1396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:18.313591 1396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:18.313632 1396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:43:18.315301 1396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:18.315538 1396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:18.315721 1396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:18.321089 1401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:18.321552 1402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:18.322110 1404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:18.322961 1396 server_base.cc:1034] running on GCE node
I20240506 15:43:18.323242 1396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:18.323515 1396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:18.324685 1396 hybrid_clock.cc:648] HybridClock initialized: now 1715010198324671 us; error 36 us; skew 500 ppm
May 06 15:43:18 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:18.327841 1396 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:43:18.328352 1396 webserver.cc:460] Webserver started at http://127.17.240.193:34495/ using document root <none> and password file <none>
I20240506 15:43:18.328553 1396 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:18.328603 1396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:18.328714 1396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:43:18.329964 1396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "590edb5b06bf4d2095bf5a1899592fb7"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "beb0a05d02c82e56982428c98ff34b02"
server_key_iv: "42fd263e47b756d66003104459a62e10"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.330300 1396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "590edb5b06bf4d2095bf5a1899592fb7"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "beb0a05d02c82e56982428c98ff34b02"
server_key_iv: "42fd263e47b756d66003104459a62e10"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.331768 1396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:43:18.332558 1411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:18.332724 1396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:43:18.332808 1396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "590edb5b06bf4d2095bf5a1899592fb7"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "beb0a05d02c82e56982428c98ff34b02"
server_key_iv: "42fd263e47b756d66003104459a62e10"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.332871 1396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:18.356246 1396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:18.358376 1396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:18.358511 1396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:18.358698 1396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:18.358954 1396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:18.358986 1396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:18.359030 1396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:18.359055 1396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:18.364729 1396 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:37411
I20240506 15:43:18.364823 1524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:37411 every 8 connection(s)
I20240506 15:43:18.365096 1396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:43:18.366138 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 1396
I20240506 15:43:18.366235 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:43:18.366426 18371 external_mini_cluster.cc:1428] Setting key 949a8a7728e2047cb20e02e3a5d96128
May 06 15:43:18 dist-test-slave-v0k1 krb5kdc[1289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
I20240506 15:43:18.374658 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.365811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:56531 (local address 127.17.240.254:38131)
0506 15:43:18.365895 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.365897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.366059 (+ 162us) server_negotiation.cc:407] Connection header received
0506 15:43:18.366620 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.366621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.366635 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.366687 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:18.367836 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.368610 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.369155 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.369329 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.371554 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:18.371558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:18.371559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:18.371565 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:18.373141 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:18.373638 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:18.373639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:18.373639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:18.373668 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:18.373902 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:18.373903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:18.373903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:18.373999 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:18.374048 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.374456 (+ 408us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.374562 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240506 15:43:18.374740 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.365768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:56531)
0506 15:43:18.365924 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:18.365955 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:18.366499 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:18.366756 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:18.366759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:18.367109 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:18.367730 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.367735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.368693 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.368694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:18.369071 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.369073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.369212 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.369810 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:18.369826 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:18.371424 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:18.373235 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:18.373243 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:18.373246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:18.373549 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:18.373739 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:18.373740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:18.373740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:18.373836 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:18.374061 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:18.374066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:18.374227 (+ 161us) client_negotiation.cc:769] Sending connection context
0506 15:43:18.374359 (+ 132us) client_negotiation.cc:240] Negotiation successful
0506 15:43:18.374514 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20240506 15:43:18.375313 1525 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38131
I20240506 15:43:18.375409 1525 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:18.375702 1525 heartbeater.cc:511] Master 127.17.240.254:38131 requested a full tablet report, sending...
I20240506 15:43:18.376194 1334 ts_manager.cc:194] Registered new tserver with Master: 590edb5b06bf4d2095bf5a1899592fb7 (127.17.240.193:37411)
I20240506 15:43:18.376896 1334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:56531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:43:18.394204 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38131
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:18.478016 1530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:18.478255 1530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:18.478296 1530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:18.479740 1530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:18.479797 1530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:18.479840 1530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:43:18.481535 1530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:18.481786 1530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:18.481986 1530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:18.487255 1536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:18.487578 1535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:18.488186 1538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:18.488824 1530 server_base.cc:1034] running on GCE node
I20240506 15:43:18.489007 1530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:18.489238 1530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:18.490377 1530 hybrid_clock.cc:648] HybridClock initialized: now 1715010198490358 us; error 36 us; skew 500 ppm
May 06 15:43:18 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:18.492887 1530 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:43:18.493263 1530 webserver.cc:460] Webserver started at http://127.17.240.194:39057/ using document root <none> and password file <none>
I20240506 15:43:18.493418 1530 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:18.493453 1530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:18.493546 1530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:43:18.494477 1530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "2b66e0f2a22943fdbbe44b74832f622a"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "69b58dbe3086532341a27380875fed76"
server_key_iv: "c094e11cf09457e1c327f1a11b74eb40"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.494766 1530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "2b66e0f2a22943fdbbe44b74832f622a"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "69b58dbe3086532341a27380875fed76"
server_key_iv: "c094e11cf09457e1c327f1a11b74eb40"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.496099 1530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:43:18.496927 1545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:18.497121 1530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:18.497201 1530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "2b66e0f2a22943fdbbe44b74832f622a"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "69b58dbe3086532341a27380875fed76"
server_key_iv: "c094e11cf09457e1c327f1a11b74eb40"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.497251 1530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:18.521051 1530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:18.523190 1530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:18.523291 1530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:18.523473 1530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:18.523761 1530 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:18.523797 1530 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:18.523820 1530 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:18.523847 1530 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:18.529006 1530 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:44289
I20240506 15:43:18.529080 1658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:44289 every 8 connection(s)
I20240506 15:43:18.529326 1530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
May 06 15:43:18 dist-test-slave-v0k1 krb5kdc[1289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:18.537343 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.530005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:41163 (local address 127.17.240.254:38131)
0506 15:43:18.530073 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.530075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.530229 (+ 154us) server_negotiation.cc:407] Connection header received
0506 15:43:18.530706 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.530707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.530718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.530776 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:18.531768 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.532256 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.532775 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.532899 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.534782 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:18.534786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:18.534786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:18.534793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:18.536206 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:18.536566 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:18.536567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:18.536567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:18.536593 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:18.536765 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:18.536767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:18.536767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:18.536858 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:18.536916 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.537152 (+ 236us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.537244 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240506 15:43:18.537473 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.529964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:41163)
0506 15:43:18.530121 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:18.530150 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:18.530621 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:18.530829 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:18.530832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:18.531124 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:18.531689 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.531693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.532321 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.532322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:18.532698 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.532699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.532802 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.533300 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:18.533313 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:18.534687 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:18.536281 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:18.536289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:18.536293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:18.536497 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:18.536648 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:18.536649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:18.536649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:18.536709 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:18.536912 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:18.536917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:18.537031 (+ 114us) client_negotiation.cc:769] Sending connection context
0506 15:43:18.537128 (+ 97us) client_negotiation.cc:240] Negotiation successful
0506 15:43:18.537285 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20240506 15:43:18.537905 1659 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38131
I20240506 15:43:18.537994 1659 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:18.538175 1659 heartbeater.cc:511] Master 127.17.240.254:38131 requested a full tablet report, sending...
I20240506 15:43:18.538552 1334 ts_manager.cc:194] Registered new tserver with Master: 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289)
I20240506 15:43:18.538971 1334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:41163
I20240506 15:43:18.539316 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 1530
I20240506 15:43:18.539415 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:43:18.539634 18371 external_mini_cluster.cc:1428] Setting key 439fa7941aac79096b8859aaad75c75c
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:43:18.563903 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38131
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:18.643180 1664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:18.643374 1664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:18.643397 1664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:18.644685 1664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:18.644724 1664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:18.644749 1664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:43:18.646293 1664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:18.646462 1664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:18.646615 1664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:18.651489 1669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:18.652164 1672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:18.652732 1670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:18.653324 1664 server_base.cc:1034] running on GCE node
I20240506 15:43:18.653507 1664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:18.653689 1664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:18.654819 1664 hybrid_clock.cc:648] HybridClock initialized: now 1715010198654803 us; error 33 us; skew 500 ppm
May 06 15:43:18 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:18.657232 1664 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:43:18.657635 1664 webserver.cc:460] Webserver started at http://127.17.240.195:41997/ using document root <none> and password file <none>
I20240506 15:43:18.657806 1664 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:18.657856 1664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:18.657963 1664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:43:18.658892 1664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "2b18683f46a346f39dd0a44fb7278631"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "3a08631b6289121b6109bb665eee80fb"
server_key_iv: "f237c3b285704f0af0bac07454e06e13"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.659206 1664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "2b18683f46a346f39dd0a44fb7278631"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "3a08631b6289121b6109bb665eee80fb"
server_key_iv: "f237c3b285704f0af0bac07454e06e13"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.660302 1664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:18.660928 1679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:18.661109 1664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:43:18.661183 1664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "2b18683f46a346f39dd0a44fb7278631"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "3a08631b6289121b6109bb665eee80fb"
server_key_iv: "f237c3b285704f0af0bac07454e06e13"
server_key_version: "encryptionkey@0"
I20240506 15:43:18.661243 1664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:18.678119 1664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:18.680346 1664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:18.680475 1664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:18.680728 1664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:18.681041 1664 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:18.681077 1664 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:18.681111 1664 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:18.681143 1664 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:18.687100 1664 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:33727
I20240506 15:43:18.687165 1792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:33727 every 8 connection(s)
I20240506 15:43:18.687471 1664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:43:18.688402 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 1664
I20240506 15:43:18.688544 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:43:18.688738 18371 external_mini_cluster.cc:1428] Setting key 1022493148a338314b23914c74c4aad1
May 06 15:43:18 dist-test-slave-v0k1 krb5kdc[1289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:18.696012 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.688192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:50267 (local address 127.17.240.254:38131)
0506 15:43:18.688262 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.688264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.688442 (+ 178us) server_negotiation.cc:407] Connection header received
0506 15:43:18.689074 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.689075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.689084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.689155 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:18.690299 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.690784 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.691352 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.691521 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.693430 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:18.693434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:18.693434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:18.693441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:18.694752 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:18.695161 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:18.695162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:18.695163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:18.695191 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:18.695371 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:18.695372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:18.695372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:18.695472 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:18.695533 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.695790 (+ 257us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.695894 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:43:18.696098 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.688163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:50267)
0506 15:43:18.688327 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:18.688357 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:18.688896 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:18.689239 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:18.689243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:18.689587 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:18.690187 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.690192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.690872 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.690873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:18.691260 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.691262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.691415 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.691987 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:18.692003 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:18.693337 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:18.694831 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:18.694836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:18.694839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:18.695097 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:18.695246 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:18.695247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:18.695247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:18.695319 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:18.695529 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:18.695536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:18.695665 (+ 129us) client_negotiation.cc:769] Sending connection context
0506 15:43:18.695766 (+ 101us) client_negotiation.cc:240] Negotiation successful
0506 15:43:18.695908 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20240506 15:43:18.696599 1793 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38131
I20240506 15:43:18.696691 1793 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:18.696907 1793 heartbeater.cc:511] Master 127.17.240.254:38131 requested a full tablet report, sending...
I20240506 15:43:18.697319 1334 ts_manager.cc:194] Registered new tserver with Master: 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727)
I20240506 15:43:18.697706 1334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:50267
I20240506 15:43:18.700547 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:43:18.707419 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.701816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.17.240.254:38131)
0506 15:43:18.701878 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.701880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.701953 (+ 73us) server_negotiation.cc:407] Connection header received
0506 15:43:18.702034 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.702035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.702046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.702092 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:18.702918 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.703439 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.703996 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.704184 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.705143 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:18.705146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:18.705147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:18.705153 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:18.706396 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:18.706702 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:18.706703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:18.706703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:18.706727 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:18.706857 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:18.706858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:18.706859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:18.706980 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:18.707098 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.707217 (+ 119us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.707310 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:43:18.709460 1334 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:43:18.711127 1334 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:43:18.716864 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.713354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37411 (local address 127.0.0.1:39634)
0506 15:43:18.713526 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:18.713537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:18.713619 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:18.713822 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:18.713825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:18.713827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:18.714092 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.714095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.715520 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.715521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:18.716466 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.716468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.716550 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.716591 (+ 41us) client_negotiation.cc:769] Sending connection context
0506 15:43:18.716649 (+ 58us) client_negotiation.cc:240] Negotiation successful
0506 15:43:18.716695 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20240506 15:43:18.716874 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.713345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:33727 (local address 127.0.0.1:42296)
0506 15:43:18.713463 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:18.713509 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:18.713582 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:18.714046 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:18.714047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:18.714049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:18.714266 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.714269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.715473 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.715474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:18.716466 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.716468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.716547 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.716591 (+ 44us) client_negotiation.cc:769] Sending connection context
0506 15:43:18.716649 (+ 58us) client_negotiation.cc:240] Negotiation successful
0506 15:43:18.716695 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":73,"threads_started":1}
I20240506 15:43:18.717144 1807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.713585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:44289 (local address 127.0.0.1:34576)
0506 15:43:18.713701 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:18.713710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:18.713752 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:18.714407 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:18.714408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:18.714410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:18.714621 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.714623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.715831 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.715832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:18.716619 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.716621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.716687 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.716691 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:43:18.716722 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:43:18.716770 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":83,"threads_started":1}
I20240506 15:43:18.717376 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.713489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.17.240.193:37411)
0506 15:43:18.713657 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.713671 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.713681 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:43:18.713718 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.713719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.713732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.713806 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:18.714185 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.715401 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.716572 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.717151 (+ 579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.717205 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.717239 (+ 34us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.717281 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20240506 15:43:18.717929 1459 tablet_service.cc:1467] Processing CreateTablet for tablet a530f12ae6eb4f49a8f23908fd2830b8 (DEFAULT_TABLE table=test-table [id=1dcea9db460c4ca58702ed3ed09e4eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:18.718061 1809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.713990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.17.240.194:44289)
0506 15:43:18.714238 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.714252 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.714262 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:43:18.714296 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.714298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.714315 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.714402 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:18.714703 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.715752 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.716865 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.717861 (+ 996us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.717910 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.717943 (+ 33us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.717985 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20240506 15:43:18.718256 1459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a530f12ae6eb4f49a8f23908fd2830b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:18.718389 1808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.713745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.17.240.195:33727)
0506 15:43:18.713894 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.713902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.713911 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:43:18.713950 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.713952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.713966 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.714035 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:18.714574 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.715393 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.717654 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.718211 (+ 557us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.718254 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.718285 (+ 31us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.718329 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20240506 15:43:18.718609 1593 tablet_service.cc:1467] Processing CreateTablet for tablet a530f12ae6eb4f49a8f23908fd2830b8 (DEFAULT_TABLE table=test-table [id=1dcea9db460c4ca58702ed3ed09e4eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:18.718879 1593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a530f12ae6eb4f49a8f23908fd2830b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:18.721941 1727 tablet_service.cc:1467] Processing CreateTablet for tablet a530f12ae6eb4f49a8f23908fd2830b8 (DEFAULT_TABLE table=test-table [id=1dcea9db460c4ca58702ed3ed09e4eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:18.722041 1811 tablet_bootstrap.cc:492] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: Bootstrap starting.
I20240506 15:43:18.722240 1727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a530f12ae6eb4f49a8f23908fd2830b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:18.722605 1811 tablet_bootstrap.cc:654] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:18.722842 1811 log.cc:824] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:18.723177 1810 tablet_bootstrap.cc:492] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Bootstrap starting.
I20240506 15:43:18.723480 1811 tablet_bootstrap.cc:492] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: No bootstrap required, opened a new log
I20240506 15:43:18.723541 1811 ts_tablet_manager.cc:1367] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:43:18.723851 1810 tablet_bootstrap.cc:654] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:18.724172 1810 log.cc:824] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:18.724925 1810 tablet_bootstrap.cc:492] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: No bootstrap required, opened a new log
I20240506 15:43:18.725010 1810 ts_tablet_manager.cc:1367] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:43:18.725195 1814 tablet_bootstrap.cc:492] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: Bootstrap starting.
I20240506 15:43:18.725766 1814 tablet_bootstrap.cc:654] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:18.725924 1811 raft_consensus.cc:348] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:18.726084 1811 raft_consensus.cc:374] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:18.726090 1814 log.cc:824] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:18.726114 1811 raft_consensus.cc:729] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b66e0f2a22943fdbbe44b74832f622a, State: Initialized, Role: FOLLOWER
I20240506 15:43:18.726234 1811 consensus_queue.cc:260] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:18.726539 1811 ts_tablet_manager.cc:1398] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:43:18.726666 1659 heartbeater.cc:503] Master 127.17.240.254:38131 was elected leader, sending a full tablet report...
I20240506 15:43:18.726804 1814 tablet_bootstrap.cc:492] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: No bootstrap required, opened a new log
I20240506 15:43:18.726856 1814 ts_tablet_manager.cc:1367] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:43:18.727344 1810 raft_consensus.cc:348] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:18.727485 1810 raft_consensus.cc:374] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:18.727659 1810 raft_consensus.cc:729] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590edb5b06bf4d2095bf5a1899592fb7, State: Initialized, Role: FOLLOWER
I20240506 15:43:18.727785 1810 consensus_queue.cc:260] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:18.728085 1525 heartbeater.cc:503] Master 127.17.240.254:38131 was elected leader, sending a full tablet report...
I20240506 15:43:18.728092 1810 ts_tablet_manager.cc:1398] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:43:18.728986 1814 raft_consensus.cc:348] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:18.729121 1814 raft_consensus.cc:374] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:18.729148 1814 raft_consensus.cc:729] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b18683f46a346f39dd0a44fb7278631, State: Initialized, Role: FOLLOWER
I20240506 15:43:18.729233 1814 consensus_queue.cc:260] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:18.729470 1814 ts_tablet_manager.cc:1398] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:43:18.729507 1793 heartbeater.cc:503] Master 127.17.240.254:38131 was elected leader, sending a full tablet report...
W20240506 15:43:18.779887 1660 tablet.cc:2369] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:43:18.787645 1817 raft_consensus.cc:482] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:43:18.787771 1817 raft_consensus.cc:504] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:18.788141 1817 leader_election.cc:288] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727), 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289)
I20240506 15:43:18.790383 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.788318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:33727 (local address 127.17.240.193:52573)
0506 15:43:18.788377 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:18.788390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:18.788436 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:18.788653 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:18.788654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:18.788656 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:18.788846 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.788847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.789519 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.789521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:18.790155 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.790157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.790228 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.790232 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:43:18.790264 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:43:18.790303 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20240506 15:43:18.790762 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.788428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:44289 (local address 127.17.240.193:54705)
0506 15:43:18.788649 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:18.788656 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:18.788711 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:18.788848 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:18.788849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:18.788851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:18.789101 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.789103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.789781 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.789782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:18.790494 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:18.790496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.790614 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.790618 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:43:18.790650 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:43:18.790686 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":131,"threads_started":1}
I20240506 15:43:18.790896 1808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.788408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:52573 (local address 127.17.240.195:33727)
0506 15:43:18.788496 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.788498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.788509 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:43:18.788547 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.788548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.788558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.788634 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:18.788927 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.789436 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.790238 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.790746 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.790755 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.790778 (+ 23us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.790811 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20240506 15:43:18.791319 1809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.788517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:54705 (local address 127.17.240.194:44289)
0506 15:43:18.788588 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.788590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.788717 (+ 127us) server_negotiation.cc:407] Connection header received
0506 15:43:18.788764 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.788767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.788775 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.788820 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:18.789176 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.789680 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.790645 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.791163 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.791170 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.791197 (+ 27us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.791229 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240506 15:43:18.791255 1747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a530f12ae6eb4f49a8f23908fd2830b8" candidate_uuid: "590edb5b06bf4d2095bf5a1899592fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b18683f46a346f39dd0a44fb7278631" is_pre_election: true
I20240506 15:43:18.791440 1747 raft_consensus.cc:2454] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590edb5b06bf4d2095bf5a1899592fb7 in term 0.
I20240506 15:43:18.791673 1413 leader_election.cc:302] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b18683f46a346f39dd0a44fb7278631, 590edb5b06bf4d2095bf5a1899592fb7; no voters:
I20240506 15:43:18.791674 1613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a530f12ae6eb4f49a8f23908fd2830b8" candidate_uuid: "590edb5b06bf4d2095bf5a1899592fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b66e0f2a22943fdbbe44b74832f622a" is_pre_election: true
I20240506 15:43:18.791810 1613 raft_consensus.cc:2454] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590edb5b06bf4d2095bf5a1899592fb7 in term 0.
I20240506 15:43:18.791838 1817 raft_consensus.cc:2781] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:43:18.791890 1817 raft_consensus.cc:482] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:43:18.791922 1817 raft_consensus.cc:3037] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:18.792625 1817 raft_consensus.cc:504] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:18.792748 1817 leader_election.cc:288] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 1 election: Requested vote from peers 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727), 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289)
I20240506 15:43:18.793005 1613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a530f12ae6eb4f49a8f23908fd2830b8" candidate_uuid: "590edb5b06bf4d2095bf5a1899592fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b66e0f2a22943fdbbe44b74832f622a"
I20240506 15:43:18.793082 1613 raft_consensus.cc:3037] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:18.793830 1613 raft_consensus.cc:2454] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590edb5b06bf4d2095bf5a1899592fb7 in term 1.
I20240506 15:43:18.793946 1747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a530f12ae6eb4f49a8f23908fd2830b8" candidate_uuid: "590edb5b06bf4d2095bf5a1899592fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b18683f46a346f39dd0a44fb7278631"
I20240506 15:43:18.794034 1413 leader_election.cc:302] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b66e0f2a22943fdbbe44b74832f622a, 590edb5b06bf4d2095bf5a1899592fb7; no voters:
I20240506 15:43:18.794044 1747 raft_consensus.cc:3037] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:18.794154 1817 raft_consensus.cc:2781] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:43:18.794343 1817 raft_consensus.cc:686] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 LEADER]: Becoming Leader. State: Replica: 590edb5b06bf4d2095bf5a1899592fb7, State: Running, Role: LEADER
I20240506 15:43:18.794472 1817 consensus_queue.cc:237] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:18.794991 1747 raft_consensus.cc:2454] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590edb5b06bf4d2095bf5a1899592fb7 in term 1.
I20240506 15:43:18.795416 1334 catalog_manager.cc:5484] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 590edb5b06bf4d2095bf5a1899592fb7 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "590edb5b06bf4d2095bf5a1899592fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } health_report { overall_health: UNKNOWN } } }
I20240506 15:43:18.810124 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.808077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.17.240.193:37411)
0506 15:43:18.808142 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:18.808144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:18.808153 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:43:18.808193 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:18.808193 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:18.808210 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:18.808277 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:43:18.808527 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.809011 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:18.809576 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:18.809729 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:18.809769 (+ 40us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:43:18.809854 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:43:18.809927 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:18.810008 (+ 81us) server_negotiation.cc:299] Negotiation successful
0506 15:43:18.810033 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:43:18.811937 1613 raft_consensus.cc:1261] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a [term 1 FOLLOWER]: Refusing update from remote peer 590edb5b06bf4d2095bf5a1899592fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:43:18.811937 1747 raft_consensus.cc:1261] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631 [term 1 FOLLOWER]: Refusing update from remote peer 590edb5b06bf4d2095bf5a1899592fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:43:18.812356 1817 consensus_queue.cc:1035] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:43:18.812563 1820 consensus_queue.cc:1035] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:43:18.815567 1825 mvcc.cc:205] Tried to move back new op lower bound from 7024681774330990592 to 7024681774262964224. Current Snapshot: MvccSnapshot[applied={T|T < 7024681774330990592 or (T in {7024681774330990592})}]
I20240506 15:43:18.817222 1333 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34014:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:43:18.817312 1333 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34014:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:43:18.818074 1333 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Sending DeleteTablet for 3 replicas of tablet a530f12ae6eb4f49a8f23908fd2830b8
I20240506 15:43:18.818475 1459 tablet_service.cc:1514] Processing DeleteTablet for tablet a530f12ae6eb4f49a8f23908fd2830b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:18 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:39634
I20240506 15:43:18.818472 1593 tablet_service.cc:1514] Processing DeleteTablet for tablet a530f12ae6eb4f49a8f23908fd2830b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:18 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:34576
I20240506 15:43:18.818480 1727 tablet_service.cc:1514] Processing DeleteTablet for tablet a530f12ae6eb4f49a8f23908fd2830b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:18 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:42296
I20240506 15:43:18.818732 1835 tablet_replica.cc:330] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: stopping tablet replica
I20240506 15:43:18.818812 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 1396
I20240506 15:43:18.818845 1835 raft_consensus.cc:2229] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:43:18.818948 1835 raft_consensus.cc:2258] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:18.819654 1834 tablet_replica.cc:330] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: stopping tablet replica
I20240506 15:43:18.819788 1834 raft_consensus.cc:2229] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:43:18.819932 1834 raft_consensus.cc:2258] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:18.820000 1835 ts_tablet_manager.cc:1875] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:43:18.820452 1834 ts_tablet_manager.cc:1875] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:43:18.821782 1835 ts_tablet_manager.cc:1888] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:43:18.821868 1835 log.cc:1196] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/a530f12ae6eb4f49a8f23908fd2830b8
I20240506 15:43:18.822168 1835 ts_tablet_manager.cc:1909] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b18683f46a346f39dd0a44fb7278631: Deleting consensus metadata
I20240506 15:43:18.822361 1834 ts_tablet_manager.cc:1888] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:43:18.822479 1834 log.cc:1196] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/a530f12ae6eb4f49a8f23908fd2830b8
I20240506 15:43:18.822659 1319 catalog_manager.cc:4830] TS 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727): tablet a530f12ae6eb4f49a8f23908fd2830b8 (table test-table [id=1dcea9db460c4ca58702ed3ed09e4eed]) successfully deleted
I20240506 15:43:18.822772 1834 ts_tablet_manager.cc:1909] T a530f12ae6eb4f49a8f23908fd2830b8 P 2b66e0f2a22943fdbbe44b74832f622a: Deleting consensus metadata
I20240506 15:43:18.823163 1319 catalog_manager.cc:4830] TS 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289): tablet a530f12ae6eb4f49a8f23908fd2830b8 (table test-table [id=1dcea9db460c4ca58702ed3ed09e4eed]) successfully deleted
W20240506 15:43:18.824370 1320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:37411 (error 108)
I20240506 15:43:18.824600 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 1530
I20240506 15:43:18.825599 1807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:18.825442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37411 (local address 127.0.0.1:39658)
0506 15:43:18.825512 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:18.825555 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20240506 15:43:18.825728 1320 catalog_manager.cc:4573] TS 590edb5b06bf4d2095bf5a1899592fb7 (127.17.240.193:37411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a530f12ae6eb4f49a8f23908fd2830b8: Network error: Client connection negotiation failed: client connection to 127.17.240.193:37411: connect: Connection refused (error 111)
I20240506 15:43:18.830529 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 1664
I20240506 15:43:18.835563 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 1302
I20240506 15:43:38.841609 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38131
--webserver_interface=127.17.240.254
--webserver_port=39677
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:38131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:38.922730 1837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:38.922955 1837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:38.922977 1837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:38.924341 1837 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240506 15:43:38.924386 1837 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240506 15:43:38.924398 1837 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240506 15:43:38.924410 1837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:43:38.924435 1837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:38.924465 1837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:38.924494 1837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:43:38.924517 1837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:43:38.926205 1837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:38131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=39677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:38.926396 1837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:38.926614 1837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:38.935202 1846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:38.935848 1844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:38.936550 1843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:38.937254 1837 server_base.cc:1034] running on GCE node
I20240506 15:43:38.937500 1837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:38.937755 1837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:38.938922 1837 hybrid_clock.cc:648] HybridClock initialized: now 1715010218938906 us; error 43 us; skew 500 ppm
May 06 15:43:38 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010218, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:38.948056 1837 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:43:38.948470 1837 webserver.cc:460] Webserver started at http://127.17.240.254:39677/ using document root <none> and password file <none>
I20240506 15:43:38.948648 1837 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:38.948693 1837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:38.949988 1837 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:38.950579 1853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:38.950757 1837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:38.950832 1837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "1471f9f6ca1a49c08ffe1f73833675b9"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "c08e45a5febaa03f60f047d89657c67d"
server_key_iv: "867721989ed59a73de7253f705ac200d"
server_key_version: "encryptionkey@0"
I20240506 15:43:38.951234 1837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:38.960812 1837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:38.962903 1837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:38.963042 1837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:38.967375 1837 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:38131
I20240506 15:43:38.967403 1905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:38131 every 8 connection(s)
I20240506 15:43:38.967772 1837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:43:38.969156 1906 sys_catalog.cc:263] Verifying existing consensus state
I20240506 15:43:38.969720 1906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Bootstrap starting.
I20240506 15:43:38.976547 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 1837
I20240506 15:43:38.976805 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:37411
--local_ip_for_outbound_sockets=127.17.240.193
--tserver_master_addrs=127.17.240.254:38131
--webserver_port=34495
--webserver_interface=127.17.240.193
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:43:38.978935 1906 log.cc:824] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:38.980743 1906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:43:38.981027 1906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Bootstrap complete.
I20240506 15:43:38.983265 1906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:38.983469 1906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1471f9f6ca1a49c08ffe1f73833675b9, State: Initialized, Role: FOLLOWER
I20240506 15:43:38.983594 1906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:38.983660 1906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:43:38.983701 1906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:43:38.983750 1906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:43:38.984654 1906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:38.984755 1906 leader_election.cc:302] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1471f9f6ca1a49c08ffe1f73833675b9; no voters:
I20240506 15:43:38.984910 1906 leader_election.cc:288] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240506 15:43:38.984982 1910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:43:38.985189 1910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 2 LEADER]: Becoming Leader. State: Replica: 1471f9f6ca1a49c08ffe1f73833675b9, State: Running, Role: LEADER
I20240506 15:43:38.985217 1906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:43:38.985280 1910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:38.985631 1910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1471f9f6ca1a49c08ffe1f73833675b9. Latest consensus state: current_term: 2 leader_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } } }
I20240506 15:43:38.985656 1911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } } }
I20240506 15:43:38.985808 1910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: This master's current role is: LEADER
I20240506 15:43:38.985828 1911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: This master's current role is: LEADER
I20240506 15:43:38.986197 1919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:43:38.986996 1919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:43:38.987253 1919 catalog_manager.cc:1259] Loaded cluster ID: fb592c99d5f2449fbc1f3c0a09779253
I20240506 15:43:38.987335 1919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:43:38.988234 1919 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:43:38.988402 1919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Loaded TSK: 0
I20240506 15:43:38.994490 1919 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Generated new TSK 1
I20240506 15:43:38.994589 1919 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240506 15:43:39.068420 1908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:39.068629 1908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:39.068671 1908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:39.070081 1908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:39.070130 1908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:39.070168 1908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:43:39.071920 1908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:37411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=34495
--tserver_master_addrs=127.17.240.254:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:39.072135 1908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:39.072315 1908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:39.077885 1936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:39.078316 1933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:39.079157 1934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:39.079845 1908 server_base.cc:1034] running on GCE node
I20240506 15:43:39.080051 1908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:39.080292 1908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:39.081466 1908 hybrid_clock.cc:648] HybridClock initialized: now 1715010219081451 us; error 29 us; skew 500 ppm
May 06 15:43:39 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:39.084350 1908 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:43:39.084846 1908 webserver.cc:460] Webserver started at http://127.17.240.193:34495/ using document root <none> and password file <none>
I20240506 15:43:39.085037 1908 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:39.085094 1908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:39.086558 1908 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:39.087268 1943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:39.087442 1908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:39.087517 1908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "590edb5b06bf4d2095bf5a1899592fb7"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "beb0a05d02c82e56982428c98ff34b02"
server_key_iv: "42fd263e47b756d66003104459a62e10"
server_key_version: "encryptionkey@0"
I20240506 15:43:39.087800 1908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:39.102758 1908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:39.104866 1908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:39.104992 1908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:39.105188 1908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:39.105592 1950 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240506 15:43:39.106716 1908 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 15:43:39.106757 1908 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:39.106794 1908 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 15:43:39.107669 1908 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 15:43:39.107707 1908 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:39.107790 1950 tablet_bootstrap.cc:492] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Bootstrap starting.
I20240506 15:43:39.114646 1908 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:37411
I20240506 15:43:39.114811 2057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:37411 every 8 connection(s)
I20240506 15:43:39.115061 1908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:43:39.120440 1950 log.cc:824] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:39.121842 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 1908
I20240506 15:43:39.122138 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:44289
--local_ip_for_outbound_sockets=127.17.240.194
--tserver_master_addrs=127.17.240.254:38131
--webserver_port=39057
--webserver_interface=127.17.240.194
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:43:39.123149 1950 tablet_bootstrap.cc:492] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:43:39.123456 1950 tablet_bootstrap.cc:492] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Bootstrap complete.
I20240506 15:43:39.125104 1950 ts_tablet_manager.cc:1367] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.005s
May 06 15:43:39 dist-test-slave-v0k1 krb5kdc[1289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:39.126366 1950 raft_consensus.cc:348] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:39.126665 1950 raft_consensus.cc:729] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590edb5b06bf4d2095bf5a1899592fb7, State: Initialized, Role: FOLLOWER
I20240506 15:43:39.126792 1950 consensus_queue.cc:260] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } } peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } }
I20240506 15:43:39.127074 1950 ts_tablet_manager.cc:1398] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:43:39.135694 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.115744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:57181)
0506 15:43:39.115928 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:39.115959 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:39.116400 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:39.121115 (+ 4715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:39.121121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:39.121438 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:39.121996 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.122004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.123666 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.123668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:39.124349 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.124352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.124452 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.125118 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:39.125134 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:39.126679 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:39.133850 (+ 7171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:39.133860 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:39.133863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:39.134190 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:39.134471 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:39.134472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:39.134472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:39.134568 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:39.135053 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:39.135060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:39.135210 (+ 150us) client_negotiation.cc:769] Sending connection context
0506 15:43:39.135339 (+ 129us) client_negotiation.cc:240] Negotiation successful
0506 15:43:39.135507 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20240506 15:43:39.136087 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.115817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:57181 (local address 127.17.240.254:38131)
0506 15:43:39.120651 (+ 4834us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.120663 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.120696 (+ 33us) server_negotiation.cc:407] Connection header received
0506 15:43:39.120824 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.120830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.120886 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.121076 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:39.122134 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.123548 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.124707 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.124885 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.131142 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:39.131170 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:39.131170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:39.131198 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:39.133705 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.134297 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.134300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.134300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.134372 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.134647 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.134649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.134649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.134909 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:39.135002 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.135429 (+ 427us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.135571 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4786,"thread_start_us":89,"threads_started":1}
I20240506 15:43:39.136795 2058 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38131
I20240506 15:43:39.136911 2058 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:39.137133 2058 heartbeater.cc:511] Master 127.17.240.254:38131 requested a full tablet report, sending...
I20240506 15:43:39.137946 1870 ts_manager.cc:194] Registered new tserver with Master: 590edb5b06bf4d2095bf5a1899592fb7 (127.17.240.193:37411)
I20240506 15:43:39.139205 1870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:57181
I20240506 15:43:39.140084 2058 heartbeater.cc:503] Master 127.17.240.254:38131 was elected leader, sending a full tablet report...
May 06 15:43:39 dist-test-slave-v0k1 krb5kdc[1289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010218, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.193@KRBTEST.COM
I20240506 15:43:39.148344 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.138796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.17.240.193:37411)
0506 15:43:39.139446 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.139454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.139464 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:43:39.139500 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.139502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.139517 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.139586 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:39.140614 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.141626 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.142415 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.142679 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.144961 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:39.144972 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:39.144972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:39.144979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:39.146966 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.147472 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.147475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.147476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.147517 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.147692 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.147693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.147694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.147856 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:39.147968 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.148143 (+ 175us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.148249 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":89,"threads_started":1}
I20240506 15:43:39.148505 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.138555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37411 (local address 127.0.0.1:47332)
0506 15:43:39.139129 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:39.139155 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:39.139225 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:39.139674 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:39.139679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:39.140133 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:39.140521 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.140524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.141749 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.141750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:39.142310 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.142313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.142400 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.143170 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:39.143174 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:39.144844 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:39.147135 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:39.147140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:39.147143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:39.147378 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:39.147596 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:39.147597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:39.147597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:39.147630 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:39.147940 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:39.147945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:39.148004 (+ 59us) client_negotiation.cc:769] Sending connection context
0506 15:43:39.148094 (+ 90us) client_negotiation.cc:240] Negotiation successful
0506 15:43:39.148204 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":99,"threads_started":1}
I20240506 15:43:39.149066 1992 tablet_service.cc:1514] Processing DeleteTablet for tablet a530f12ae6eb4f49a8f23908fd2830b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:18 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:47332
I20240506 15:43:39.149672 2067 tablet_replica.cc:330] stopping tablet replica
I20240506 15:43:39.149806 2067 raft_consensus.cc:2229] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:43:39.149924 2067 raft_consensus.cc:2258] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:39.150300 2067 ts_tablet_manager.cc:1875] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:43:39.152823 2067 ts_tablet_manager.cc:1888] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:43:39.152956 2067 log.cc:1196] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal/wals/a530f12ae6eb4f49a8f23908fd2830b8
I20240506 15:43:39.153281 2067 ts_tablet_manager.cc:1909] T a530f12ae6eb4f49a8f23908fd2830b8 P 590edb5b06bf4d2095bf5a1899592fb7: Deleting consensus metadata
I20240506 15:43:39.153838 1856 catalog_manager.cc:4830] TS 590edb5b06bf4d2095bf5a1899592fb7 (127.17.240.193:37411): tablet a530f12ae6eb4f49a8f23908fd2830b8 (table test-table [id=1dcea9db460c4ca58702ed3ed09e4eed]) successfully deleted
W20240506 15:43:39.213689 2063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:39.213912 2063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:39.213937 2063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:39.215268 2063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:39.215312 2063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:39.215337 2063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:43:39.216907 2063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:44289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=39057
--tserver_master_addrs=127.17.240.254:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:39.217075 2063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:39.217223 2063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:39.222133 2072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:39.222692 2073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:39.223284 2075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:39.224030 2063 server_base.cc:1034] running on GCE node
I20240506 15:43:39.224232 2063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:39.224515 2063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:39.225679 2063 hybrid_clock.cc:648] HybridClock initialized: now 1715010219225652 us; error 38 us; skew 500 ppm
May 06 15:43:39 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:39.228228 2063 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:43:39.228652 2063 webserver.cc:460] Webserver started at http://127.17.240.194:39057/ using document root <none> and password file <none>
I20240506 15:43:39.228825 2063 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:39.228884 2063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:39.230177 2063 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:39.230799 2082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:39.231001 2063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:39.231110 2063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "2b66e0f2a22943fdbbe44b74832f622a"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "69b58dbe3086532341a27380875fed76"
server_key_iv: "c094e11cf09457e1c327f1a11b74eb40"
server_key_version: "encryptionkey@0"
I20240506 15:43:39.231424 2063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:39.249652 2063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:39.251534 2063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:39.251658 2063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:39.251873 2063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:39.252167 2063 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:39.252200 2063 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:39.252231 2063 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:39.252280 2063 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:39.257818 2063 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:44289
I20240506 15:43:39.257913 2195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:44289 every 8 connection(s)
I20240506 15:43:39.258179 2063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:43:39.259631 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 2063
I20240506 15:43:39.259960 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:33727
--local_ip_for_outbound_sockets=127.17.240.195
--tserver_master_addrs=127.17.240.254:38131
--webserver_port=41997
--webserver_interface=127.17.240.195
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 06 15:43:39 dist-test-slave-v0k1 krb5kdc[1289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:39.267977 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.258870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:48773 (local address 127.17.240.254:38131)
0506 15:43:39.258933 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.258935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.259133 (+ 198us) server_negotiation.cc:407] Connection header received
0506 15:43:39.259710 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.259712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.259729 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.259807 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:39.261281 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.261787 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.262599 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.262709 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.265175 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:39.265180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:39.265180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:39.265187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:39.266629 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.267134 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.267135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.267135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.267157 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.267373 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.267373 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.267373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.267464 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:39.267510 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.267827 (+ 317us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.267896 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240506 15:43:39.268200 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.258829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:48773)
0506 15:43:39.259025 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:39.259051 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:39.259577 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:39.259890 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:39.259895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:39.260362 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:39.261174 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.261180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.261886 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.261887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:39.262521 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.262524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.262667 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.263285 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:39.263309 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:39.265068 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:39.266719 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:39.266725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:39.266729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:39.267061 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:39.267230 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:39.267230 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:39.267231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:39.267312 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:39.267525 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:39.267529 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:39.267686 (+ 157us) client_negotiation.cc:769] Sending connection context
0506 15:43:39.267810 (+ 124us) client_negotiation.cc:240] Negotiation successful
0506 15:43:39.267967 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20240506 15:43:39.268765 2196 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38131
I20240506 15:43:39.268855 2196 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:39.269054 2196 heartbeater.cc:511] Master 127.17.240.254:38131 requested a full tablet report, sending...
I20240506 15:43:39.269404 1870 ts_manager.cc:194] Registered new tserver with Master: 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289)
I20240506 15:43:39.269778 1870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:48773
W20240506 15:43:39.341761 2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:39.341979 2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:39.342015 2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:39.343428 2199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:39.343477 2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:39.343515 2199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:43:39.345115 2199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:33727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=41997
--tserver_master_addrs=127.17.240.254:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:39.345341 2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:39.345530 2199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:39.350559 2207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:39.350914 2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:39.351545 2205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:39.352246 2199 server_base.cc:1034] running on GCE node
I20240506 15:43:39.352422 2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:39.352618 2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:39.353742 2199 hybrid_clock.cc:648] HybridClock initialized: now 1715010219353726 us; error 36 us; skew 500 ppm
May 06 15:43:39 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:39.356180 2199 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:43:39.356560 2199 webserver.cc:460] Webserver started at http://127.17.240.195:41997/ using document root <none> and password file <none>
I20240506 15:43:39.356710 2199 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:39.356756 2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:39.357944 2199 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:39.358528 2214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:39.358783 2199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:39.358871 2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "2b18683f46a346f39dd0a44fb7278631"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "3a08631b6289121b6109bb665eee80fb"
server_key_iv: "f237c3b285704f0af0bac07454e06e13"
server_key_version: "encryptionkey@0"
I20240506 15:43:39.359189 2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:39.380141 2199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:39.381902 2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:39.382014 2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:39.382200 2199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:39.382450 2199 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:43:39.382480 2199 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:39.382503 2199 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:43:39.382520 2199 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:39.388008 2199 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:33727
I20240506 15:43:39.388044 2327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:33727 every 8 connection(s)
I20240506 15:43:39.388350 2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
May 06 15:43:39 dist-test-slave-v0k1 krb5kdc[1289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:43:39.395512 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 2199
I20240506 15:43:39.397408 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.388999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:47031 (local address 127.17.240.254:38131)
0506 15:43:39.389060 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.389062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.389378 (+ 316us) server_negotiation.cc:407] Connection header received
0506 15:43:39.390113 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.390115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.390127 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.390193 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:39.391527 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.392018 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.392601 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.392721 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.394623 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:39.394627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:39.394628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:39.394633 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:39.396055 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.396562 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.396563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.396563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.396584 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.396794 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.396795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.396795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.396874 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:39.396921 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.397227 (+ 306us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.397309 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240506 15:43:39.397544 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.389036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:47031)
0506 15:43:39.389208 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:39.389242 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:39.389989 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:39.390264 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:39.390269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:39.390735 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:43:39.391394 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.391400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.392105 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.392106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:39.392520 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.392522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.392634 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.393188 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:43:39.393202 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:43:39.394546 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:43:39.396143 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:39.396149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:39.396152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:39.396487 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:39.396650 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:43:39.396651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:43:39.396652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:43:39.396735 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:43:39.396935 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:43:39.396940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:43:39.397090 (+ 150us) client_negotiation.cc:769] Sending connection context
0506 15:43:39.397208 (+ 118us) client_negotiation.cc:240] Negotiation successful
0506 15:43:39.397329 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20240506 15:43:39.398051 2328 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38131
I20240506 15:43:39.398147 2328 heartbeater.cc:464] Registering TS with master...
I20240506 15:43:39.398334 2328 heartbeater.cc:511] Master 127.17.240.254:38131 requested a full tablet report, sending...
I20240506 15:43:39.399111 1870 ts_manager.cc:194] Registered new tserver with Master: 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727)
I20240506 15:43:39.399495 1870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:47031
I20240506 15:43:39.403391 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.395962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.17.240.254:38131)
0506 15:43:39.396093 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.396095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.396135 (+ 40us) server_negotiation.cc:407] Connection header received
0506 15:43:39.396213 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.396214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.396222 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.396271 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:39.397216 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.397921 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.398646 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.398828 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.400600 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:39.400605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:39.400606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:39.400613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:39.402179 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.402543 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.402544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.402544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.402569 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.402796 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.402798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.402798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.402903 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:39.403066 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.403213 (+ 147us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.403306 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20240506 15:43:39.404254 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:43:39.406515 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.404646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.17.240.254:38131)
0506 15:43:39.404718 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.404719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.404790 (+ 71us) server_negotiation.cc:407] Connection header received
0506 15:43:39.404841 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.404841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.404863 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.404936 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:43:39.405168 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.405630 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.406172 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.406274 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.406306 (+ 32us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:43:39.406345 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0506 15:43:39.406464 (+ 119us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56782: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20240506 15:43:39.406561 2332 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56782: token expired
W20240506 15:43:39.406685 1799 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.240.254:38131)
I20240506 15:43:39.411876 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.406927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.17.240.254:38131)
0506 15:43:39.406989 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.406991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.406998 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:43:39.407050 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.407051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.407062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.407118 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:43:39.407680 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.408139 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.408694 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.408809 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.409736 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:43:39.409739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:43:39.409740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:43:39.409743 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:43:39.410942 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.411281 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.411282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.411283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.411318 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:43:39.411459 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:43:39.411459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:43:39.411460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:43:39.411535 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:43:39.411607 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.411749 (+ 142us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.411814 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240506 15:43:39.413089 1870 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:43:39.414515 1870 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:43:39.416754 1992 tablet_service.cc:1467] Processing CreateTablet for tablet ae6989ee74c147a6a3d9cb2b7710713c (DEFAULT_TABLE table=test-table [id=b672d4d54b7646c68b0facfcd5e86a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:39.416977 1992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6989ee74c147a6a3d9cb2b7710713c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:39.418548 1950 tablet_bootstrap.cc:492] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: Bootstrap starting.
I20240506 15:43:39.418848 1950 tablet_bootstrap.cc:654] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:39.419387 1950 tablet_bootstrap.cc:492] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: No bootstrap required, opened a new log
I20240506 15:43:39.419438 1950 ts_tablet_manager.cc:1367] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:39.419544 1950 raft_consensus.cc:348] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:39.419616 1950 raft_consensus.cc:374] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:39.419639 1950 raft_consensus.cc:729] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590edb5b06bf4d2095bf5a1899592fb7, State: Initialized, Role: FOLLOWER
I20240506 15:43:39.419624 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.416384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:33727 (local address 127.0.0.1:51298)
0506 15:43:39.416569 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:39.416578 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:39.416626 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:39.416982 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:39.416983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:39.416985 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:39.417198 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.417200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.418300 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.418301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:39.419361 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.419365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.419419 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.419465 (+ 46us) client_negotiation.cc:769] Sending connection context
0506 15:43:39.419502 (+ 37us) client_negotiation.cc:240] Negotiation successful
0506 15:43:39.419551 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":122,"threads_started":1}
I20240506 15:43:39.419685 1950 consensus_queue.cc:260] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:39.419776 1950 ts_tablet_manager.cc:1398] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240506 15:43:39.420426 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.416382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.17.240.194:44289)
0506 15:43:39.416750 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.416758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.416768 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:43:39.416809 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.416811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.416829 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.417231 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:39.417302 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.418384 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.419679 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.420224 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.420275 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.420309 (+ 34us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.420349 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":125,"threads_started":1}
I20240506 15:43:39.420423 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.416641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.17.240.195:33727)
0506 15:43:39.416795 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.416801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.416809 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:43:39.416839 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.416841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.416858 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.416984 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:39.417275 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.418215 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.419731 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.420232 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.420275 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.420309 (+ 34us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.420348 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20240506 15:43:39.420547 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.416292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:44289 (local address 127.0.0.1:37374)
0506 15:43:39.416327 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:39.416337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:39.416369 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:39.416946 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:39.416947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:39.416949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:39.417137 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.417139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.418648 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.418649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:39.419480 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.419482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.419538 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.419541 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:43:39.419569 (+ 28us) client_negotiation.cc:240] Negotiation successful
0506 15:43:39.419598 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20240506 15:43:39.420993 2262 tablet_service.cc:1467] Processing CreateTablet for tablet ae6989ee74c147a6a3d9cb2b7710713c (DEFAULT_TABLE table=test-table [id=b672d4d54b7646c68b0facfcd5e86a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:39.421062 2130 tablet_service.cc:1467] Processing CreateTablet for tablet ae6989ee74c147a6a3d9cb2b7710713c (DEFAULT_TABLE table=test-table [id=b672d4d54b7646c68b0facfcd5e86a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:43:39.421257 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6989ee74c147a6a3d9cb2b7710713c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:39.421283 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6989ee74c147a6a3d9cb2b7710713c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:43:39.423815 2339 tablet_bootstrap.cc:492] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: Bootstrap starting.
I20240506 15:43:39.423902 2338 tablet_bootstrap.cc:492] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: Bootstrap starting.
I20240506 15:43:39.424350 2339 tablet_bootstrap.cc:654] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:39.424362 2338 tablet_bootstrap.cc:654] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: Neither blocks nor log segments found. Creating new log.
I20240506 15:43:39.424599 2338 log.cc:824] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:39.424602 2339 log.cc:824] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:39.425259 2339 tablet_bootstrap.cc:492] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: No bootstrap required, opened a new log
I20240506 15:43:39.425329 2339 ts_tablet_manager.cc:1367] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:43:39.425545 2338 tablet_bootstrap.cc:492] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: No bootstrap required, opened a new log
I20240506 15:43:39.425611 2338 ts_tablet_manager.cc:1367] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:43:39.427634 2339 raft_consensus.cc:348] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:39.427781 2339 raft_consensus.cc:374] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:39.427824 2339 raft_consensus.cc:729] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b18683f46a346f39dd0a44fb7278631, State: Initialized, Role: FOLLOWER
I20240506 15:43:39.427822 2338 raft_consensus.cc:348] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:39.427942 2338 raft_consensus.cc:374] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:43:39.427917 2339 consensus_queue.cc:260] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:39.427978 2338 raft_consensus.cc:729] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b66e0f2a22943fdbbe44b74832f622a, State: Initialized, Role: FOLLOWER
I20240506 15:43:39.428063 2338 consensus_queue.cc:260] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:39.428185 2339 ts_tablet_manager.cc:1398] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:43:39.428223 2328 heartbeater.cc:503] Master 127.17.240.254:38131 was elected leader, sending a full tablet report...
I20240506 15:43:39.428293 2338 ts_tablet_manager.cc:1398] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:43:39.428354 2196 heartbeater.cc:503] Master 127.17.240.254:38131 was elected leader, sending a full tablet report...
W20240506 15:43:39.508740 2197 tablet.cc:2369] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:43:39.557381 2064 raft_consensus.cc:482] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:43:39.557543 2064 raft_consensus.cc:504] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:39.557861 2064 leader_election.cc:288] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289), 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727)
I20240506 15:43:39.560411 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.558085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:44289 (local address 127.17.240.193:59191)
0506 15:43:39.558167 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:39.558181 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:39.558255 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:39.558440 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:39.558442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:39.558444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:39.558638 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.558640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.559544 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.559545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:39.560223 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.560225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.560284 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.560310 (+ 26us) client_negotiation.cc:769] Sending connection context
0506 15:43:39.560333 (+ 23us) client_negotiation.cc:240] Negotiation successful
0506 15:43:39.560359 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20240506 15:43:39.560789 2344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.558189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:33727 (local address 127.17.240.193:40865)
0506 15:43:39.558378 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:39.558387 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:39.558423 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:43:39.558559 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:43:39.558560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:43:39.558561 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:43:39.559099 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.559102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.559854 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.559855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:43:39.560549 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:43:39.560551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.560639 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.560642 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:43:39.560675 (+ 33us) client_negotiation.cc:240] Negotiation successful
0506 15:43:39.560708 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":112,"threads_started":1}
I20240506 15:43:39.560935 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.558215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:59191 (local address 127.17.240.194:44289)
0506 15:43:39.558285 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.558288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.558298 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:43:39.558338 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.558339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.558348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.558406 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:39.558723 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.559460 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.560321 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.560812 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.560821 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.560844 (+ 23us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.560883 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240506 15:43:39.561228 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.558338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:40865 (local address 127.17.240.195:33727)
0506 15:43:39.558405 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.558407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.558431 (+ 24us) server_negotiation.cc:407] Connection header received
0506 15:43:39.558483 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.558484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.558493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.558544 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:43:39.559187 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.559765 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.560648 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.561096 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.561104 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.561125 (+ 21us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.561160 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240506 15:43:39.561246 2150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6989ee74c147a6a3d9cb2b7710713c" candidate_uuid: "590edb5b06bf4d2095bf5a1899592fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b66e0f2a22943fdbbe44b74832f622a" is_pre_election: true
I20240506 15:43:39.561393 2150 raft_consensus.cc:2454] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590edb5b06bf4d2095bf5a1899592fb7 in term 0.
I20240506 15:43:39.561547 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6989ee74c147a6a3d9cb2b7710713c" candidate_uuid: "590edb5b06bf4d2095bf5a1899592fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b18683f46a346f39dd0a44fb7278631" is_pre_election: true
I20240506 15:43:39.561626 1945 leader_election.cc:302] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b66e0f2a22943fdbbe44b74832f622a, 590edb5b06bf4d2095bf5a1899592fb7; no voters:
I20240506 15:43:39.561684 2282 raft_consensus.cc:2454] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590edb5b06bf4d2095bf5a1899592fb7 in term 0.
I20240506 15:43:39.561745 2064 raft_consensus.cc:2781] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:43:39.561826 2064 raft_consensus.cc:482] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:43:39.561884 2064 raft_consensus.cc:3037] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:39.562630 2064 raft_consensus.cc:504] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:39.562757 2064 leader_election.cc:288] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 1 election: Requested vote from peers 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289), 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727)
I20240506 15:43:39.562939 2150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6989ee74c147a6a3d9cb2b7710713c" candidate_uuid: "590edb5b06bf4d2095bf5a1899592fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b66e0f2a22943fdbbe44b74832f622a"
I20240506 15:43:39.562986 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6989ee74c147a6a3d9cb2b7710713c" candidate_uuid: "590edb5b06bf4d2095bf5a1899592fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b18683f46a346f39dd0a44fb7278631"
I20240506 15:43:39.563051 2150 raft_consensus.cc:3037] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:39.563068 2282 raft_consensus.cc:3037] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:43:39.563927 2282 raft_consensus.cc:2454] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590edb5b06bf4d2095bf5a1899592fb7 in term 1.
I20240506 15:43:39.563927 2150 raft_consensus.cc:2454] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590edb5b06bf4d2095bf5a1899592fb7 in term 1.
I20240506 15:43:39.564144 1945 leader_election.cc:302] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b18683f46a346f39dd0a44fb7278631, 590edb5b06bf4d2095bf5a1899592fb7; no voters:
I20240506 15:43:39.564257 2064 raft_consensus.cc:2781] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:43:39.564386 2064 raft_consensus.cc:686] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 LEADER]: Becoming Leader. State: Replica: 590edb5b06bf4d2095bf5a1899592fb7, State: Running, Role: LEADER
I20240506 15:43:39.564447 2064 consensus_queue.cc:237] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:39.565229 1868 catalog_manager.cc:5484] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 590edb5b06bf4d2095bf5a1899592fb7 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "590edb5b06bf4d2095bf5a1899592fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } health_report { overall_health: HEALTHY } } }
I20240506 15:43:39.602221 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.600128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.17.240.193:37411)
0506 15:43:39.600190 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:43:39.600192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:43:39.600201 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:43:39.600264 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:43:39.600265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:43:39.600284 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:43:39.600350 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:43:39.600581 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.601034 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:43:39.601673 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:43:39.601821 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:43:39.601864 (+ 43us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:43:39.601952 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:43:39.602023 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0506 15:43:39.602130 (+ 107us) server_negotiation.cc:299] Negotiation successful
0506 15:43:39.602158 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240506 15:43:39.603689 2282 raft_consensus.cc:1261] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631 [term 1 FOLLOWER]: Refusing update from remote peer 590edb5b06bf4d2095bf5a1899592fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:43:39.603700 2150 raft_consensus.cc:1261] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a [term 1 FOLLOWER]: Refusing update from remote peer 590edb5b06bf4d2095bf5a1899592fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:43:39.604060 2064 consensus_queue.cc:1035] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:43:39.604210 2345 consensus_queue.cc:1035] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:43:39.606882 2349 mvcc.cc:205] Tried to move back new op lower bound from 7024681859494555648 to 7024681859336527872. Current Snapshot: MvccSnapshot[applied={T|T < 7024681859494555648 or (T in {7024681859494555648})}]
I20240506 15:43:39.607043 2348 mvcc.cc:205] Tried to move back new op lower bound from 7024681859494555648 to 7024681859336527872. Current Snapshot: MvccSnapshot[applied={T|T < 7024681859494555648 or (T in {7024681859494555648})}]
I20240506 15:43:39.608855 1868 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56796:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:43:39.608938 1868 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56796:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:43:39.609618 1868 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Sending DeleteTablet for 3 replicas of tablet ae6989ee74c147a6a3d9cb2b7710713c
I20240506 15:43:39.609892 1992 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6989ee74c147a6a3d9cb2b7710713c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:39 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:47332
I20240506 15:43:39.610008 2067 tablet_replica.cc:330] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: stopping tablet replica
I20240506 15:43:39.609988 2130 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6989ee74c147a6a3d9cb2b7710713c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:39 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:37374
I20240506 15:43:39.609997 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6989ee74c147a6a3d9cb2b7710713c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:43:39 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:51298
I20240506 15:43:39.610051 2067 raft_consensus.cc:2229] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:43:39.610136 2067 raft_consensus.cc:2258] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:39.610260 2357 tablet_replica.cc:330] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: stopping tablet replica
I20240506 15:43:39.610256 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 1908
I20240506 15:43:39.610354 2357 raft_consensus.cc:2229] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:43:39.610474 2357 raft_consensus.cc:2258] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:39.611058 2357 ts_tablet_manager.cc:1875] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:43:39.612197 2356 tablet_replica.cc:330] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: stopping tablet replica
I20240506 15:43:39.612329 2356 raft_consensus.cc:2229] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:43:39.612462 2356 raft_consensus.cc:2258] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:43:39.612870 2356 ts_tablet_manager.cc:1875] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:43:39.613124 2357 ts_tablet_manager.cc:1888] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:43:39.613193 2357 log.cc:1196] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/ae6989ee74c147a6a3d9cb2b7710713c
I20240506 15:43:39.613502 2357 ts_tablet_manager.cc:1909] T ae6989ee74c147a6a3d9cb2b7710713c P 2b18683f46a346f39dd0a44fb7278631: Deleting consensus metadata
I20240506 15:43:39.613931 1855 catalog_manager.cc:4830] TS 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727): tablet ae6989ee74c147a6a3d9cb2b7710713c (table test-table [id=b672d4d54b7646c68b0facfcd5e86a90]) successfully deleted
I20240506 15:43:39.614547 2356 ts_tablet_manager.cc:1888] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:43:39.614603 2356 log.cc:1196] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/ae6989ee74c147a6a3d9cb2b7710713c
I20240506 15:43:39.614835 2356 ts_tablet_manager.cc:1909] T ae6989ee74c147a6a3d9cb2b7710713c P 2b66e0f2a22943fdbbe44b74832f622a: Deleting consensus metadata
I20240506 15:43:39.615211 1855 catalog_manager.cc:4830] TS 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289): tablet ae6989ee74c147a6a3d9cb2b7710713c (table test-table [id=b672d4d54b7646c68b0facfcd5e86a90]) successfully deleted
W20240506 15:43:39.616156 1856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:37411 (error 108)
I20240506 15:43:39.616597 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 2063
I20240506 15:43:39.616757 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:43:39.616618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37411 (local address 127.0.0.1:45030)
0506 15:43:39.616682 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:39.616718 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20240506 15:43:39.617180 1856 catalog_manager.cc:4573] TS 590edb5b06bf4d2095bf5a1899592fb7 (127.17.240.193:37411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae6989ee74c147a6a3d9cb2b7710713c: Network error: Client connection negotiation failed: client connection to 127.17.240.193:37411: connect: Connection refused (error 111)
I20240506 15:43:39.621603 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 2199
I20240506 15:43:39.626178 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 1837
I20240506 15:43:59.631990 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38131
--webserver_interface=127.17.240.254
--webserver_port=39677
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:38131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:43:59.713963 2359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:59.714167 2359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:59.714191 2359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:59.715538 2359 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240506 15:43:59.715582 2359 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240506 15:43:59.715595 2359 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240506 15:43:59.715607 2359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:43:59.715620 2359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:59.715631 2359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:59.715642 2359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:43:59.715653 2359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:43:59.717373 2359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:38131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=39677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:59.717569 2359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:59.717718 2359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:59.722680 2367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:59.723146 2364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:59.723887 2365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:59.724543 2359 server_base.cc:1034] running on GCE node
I20240506 15:43:59.724803 2359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:59.725061 2359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:59.726220 2359 hybrid_clock.cc:648] HybridClock initialized: now 1715010239726198 us; error 34 us; skew 500 ppm
May 06 15:43:59 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010239, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:59.733287 2359 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:43:59.733744 2359 webserver.cc:460] Webserver started at http://127.17.240.254:39677/ using document root <none> and password file <none>
I20240506 15:43:59.733949 2359 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:59.733999 2359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:59.735354 2359 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:59.736017 2374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:59.736249 2359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:43:59.736330 2359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "1471f9f6ca1a49c08ffe1f73833675b9"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "c08e45a5febaa03f60f047d89657c67d"
server_key_iv: "867721989ed59a73de7253f705ac200d"
server_key_version: "encryptionkey@0"
I20240506 15:43:59.736656 2359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:59.756184 2359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:59.758383 2359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:59.758527 2359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:59.762419 2359 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:38131
I20240506 15:43:59.762473 2426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:38131 every 8 connection(s)
I20240506 15:43:59.762792 2359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:43:59.764295 2427 sys_catalog.cc:263] Verifying existing consensus state
I20240506 15:43:59.764923 2427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Bootstrap starting.
I20240506 15:43:59.767267 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 2359
I20240506 15:43:59.767601 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:37411
--local_ip_for_outbound_sockets=127.17.240.193
--tserver_master_addrs=127.17.240.254:38131
--webserver_port=34495
--webserver_interface=127.17.240.193
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:43:59.775249 2427 log.cc:824] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:59.777372 2427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:43:59.777649 2427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Bootstrap complete.
I20240506 15:43:59.779904 2427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:59.780107 2427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1471f9f6ca1a49c08ffe1f73833675b9, State: Initialized, Role: FOLLOWER
I20240506 15:43:59.780234 2427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:59.780298 2427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:43:59.780337 2427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:43:59.780395 2427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:43:59.781371 2427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:59.781466 2427 leader_election.cc:302] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1471f9f6ca1a49c08ffe1f73833675b9; no voters:
I20240506 15:43:59.781644 2427 leader_election.cc:288] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20240506 15:43:59.781698 2431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:43:59.781822 2431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [term 3 LEADER]: Becoming Leader. State: Replica: 1471f9f6ca1a49c08ffe1f73833675b9, State: Running, Role: LEADER
I20240506 15:43:59.781893 2427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:43:59.782009 2431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } }
I20240506 15:43:59.782377 2433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1471f9f6ca1a49c08ffe1f73833675b9. Latest consensus state: current_term: 3 leader_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } } }
I20240506 15:43:59.782495 2433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: This master's current role is: LEADER
I20240506 15:43:59.782403 2432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1471f9f6ca1a49c08ffe1f73833675b9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38131 } } }
I20240506 15:43:59.782588 2432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9 [sys.catalog]: This master's current role is: LEADER
I20240506 15:43:59.782792 2437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:43:59.783803 2437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:43:59.784019 2437 catalog_manager.cc:1259] Loaded cluster ID: fb592c99d5f2449fbc1f3c0a09779253
I20240506 15:43:59.784135 2437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:43:59.785109 2437 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:43:59.785301 2437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Loaded TSK: 0
I20240506 15:43:59.785383 2437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1471f9f6ca1a49c08ffe1f73833675b9: Loaded TSK: 1
F20240506 15:43:59.785607 2437 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7ada97bdcd google::LogMessage::Fail() at ??:0
@ 0x7f7ada97fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f7ada97b7cc google::LogMessage::Flush() at ??:0
@ 0x7f7ada97cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7ad91cd2f6 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7ad91cabf3 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7add19461e kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7add19483a kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7add194983 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f7add154627 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7add15f463 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7adba75a08 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7adba6a3a4 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7adbd8d6db start_thread at ??:0
@ 0x7f7ad9ed071f clone at ??:0
W20240506 15:43:59.874675 2429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:43:59.874884 2429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:43:59.874912 2429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:43:59.876519 2429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:43:59.876581 2429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:43:59.876611 2429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:43:59.878396 2429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:37411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=34495
--tserver_master_addrs=127.17.240.254:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:43:59.878635 2429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:43:59.878827 2429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:43:59.885066 2469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:59.885679 2471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:43:59.886382 2468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:43:59.886978 2429 server_base.cc:1034] running on GCE node
I20240506 15:43:59.887238 2429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:43:59.887507 2429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:43:59.888679 2429 hybrid_clock.cc:648] HybridClock initialized: now 1715010239888656 us; error 42 us; skew 500 ppm
May 06 15:43:59 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010239, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:43:59.891644 2429 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:43:59.892158 2429 webserver.cc:460] Webserver started at http://127.17.240.193:34495/ using document root <none> and password file <none>
I20240506 15:43:59.892369 2429 fs_manager.cc:359] Metadata directory not provided
I20240506 15:43:59.892421 2429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:43:59.893793 2429 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:59.894562 2478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:43:59.894794 2429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:43:59.894902 2429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "590edb5b06bf4d2095bf5a1899592fb7"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "beb0a05d02c82e56982428c98ff34b02"
server_key_iv: "42fd263e47b756d66003104459a62e10"
server_key_version: "encryptionkey@0"
I20240506 15:43:59.895356 2429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:43:59.919179 2429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:43:59.921615 2429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:43:59.921787 2429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:43:59.922055 2429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:43:59.922595 2485 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240506 15:43:59.923592 2429 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 15:43:59.923667 2429 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:59.923705 2429 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 15:43:59.924947 2429 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 15:43:59.925012 2429 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240506 15:43:59.925065 2485 tablet_bootstrap.cc:492] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: Bootstrap starting.
I20240506 15:43:59.932464 2429 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:37411
I20240506 15:43:59.932889 2429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:43:59.933424 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 2429
I20240506 15:43:59.933719 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:44289
--local_ip_for_outbound_sockets=127.17.240.194
--tserver_master_addrs=127.17.240.254:38131
--webserver_port=39057
--webserver_interface=127.17.240.194
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:43:59.936405 2592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:37411 every 8 connection(s)
I20240506 15:43:59.937244 2485 log.cc:824] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: Log is configured to *not* fsync() on all Append() calls
I20240506 15:43:59.940107 2485 tablet_bootstrap.cc:492] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:43:59.940469 2485 tablet_bootstrap.cc:492] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: Bootstrap complete.
I20240506 15:43:59.941732 2485 ts_tablet_manager.cc:1367] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20240506 15:43:59.943115 2485 raft_consensus.cc:348] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:59.943439 2485 raft_consensus.cc:729] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590edb5b06bf4d2095bf5a1899592fb7, State: Initialized, Role: FOLLOWER
I20240506 15:43:59.943565 2485 consensus_queue.cc:260] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:43:59.943856 2485 ts_tablet_manager.cc:1398] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20240506 15:44:00.006991 2595 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:43:59.933690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:49361)
0506 15:43:59.935189 (+ 1499us) negotiation.cc:107] Waiting for socket to connect
0506 15:43:59.935236 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0506 15:43:59.936029 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:00.006821 (+ 70792us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1448,"thread_start_us":69,"threads_started":1}
W20240506 15:44:00.007298 2593 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:38131 (0 consecutive failures): Network error: Failed to ping master at 127.17.240.254:38131: Client connection negotiation failed: client connection to 127.17.240.254:38131: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 15:44:00.007678 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:00.007545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:45473)
0506 15:44:00.007607 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:00.007649 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20240506 15:44:00.008031 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:00.007927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:53673)
0506 15:44:00.007977 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:00.008007 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20240506 15:44:00.008147 2593 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240506 15:44:00.019469 2596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:00.019666 2596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:00.019691 2596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:00.021090 2596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:00.021139 2596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:00.021167 2596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:44:00.022821 2596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:44289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=39057
--tserver_master_addrs=127.17.240.254:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:00.023093 2596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:00.023299 2596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:00.028236 2606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:00.028784 2603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:00.029371 2604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:00.029989 2596 server_base.cc:1034] running on GCE node
I20240506 15:44:00.030186 2596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:00.030504 2596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:00.031677 2596 hybrid_clock.cc:648] HybridClock initialized: now 1715010240031651 us; error 40 us; skew 500 ppm
May 06 15:44:00 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010240, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:00.034211 2596 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:44:00.034637 2596 webserver.cc:460] Webserver started at http://127.17.240.194:39057/ using document root <none> and password file <none>
I20240506 15:44:00.034817 2596 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:00.034864 2596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:00.036144 2596 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:00.036702 2613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:00.036863 2596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:00.036938 2596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "2b66e0f2a22943fdbbe44b74832f622a"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "69b58dbe3086532341a27380875fed76"
server_key_iv: "c094e11cf09457e1c327f1a11b74eb40"
server_key_version: "encryptionkey@0"
I20240506 15:44:00.037232 2596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:00.049131 2596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:00.051256 2596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:00.051414 2596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:00.051617 2596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:00.051911 2596 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:00.051944 2596 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:00.051980 2596 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:00.052006 2596 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:00.057397 2596 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:44289
I20240506 15:44:00.057435 2726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:44289 every 8 connection(s)
I20240506 15:44:00.057771 2596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:44:00.058691 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:00.058373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:58627)
0506 15:44:00.058516 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:00.058577 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
W20240506 15:44:00.058902 2727 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:38131 (0 consecutive failures): Network error: Failed to ping master at 127.17.240.254:38131: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
I20240506 15:44:00.059182 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:00.059100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:42983)
0506 15:44:00.059139 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:00.059164 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20240506 15:44:00.059423 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:00.059342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:45751)
0506 15:44:00.059385 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:00.059406 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20240506 15:44:00.059510 2727 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240506 15:44:00.061231 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 2596
I20240506 15:44:00.061515 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:33727
--local_ip_for_outbound_sockets=127.17.240.195
--tserver_master_addrs=127.17.240.254:38131
--webserver_port=41997
--webserver_interface=127.17.240.195
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:00.142769 2730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:00.142978 2730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:00.143024 2730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:00.144460 2730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:00.144515 2730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:00.144541 2730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:44:00.146310 2730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:33727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=41997
--tserver_master_addrs=127.17.240.254:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:00.146543 2730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:00.146750 2730 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:00.151939 2736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:00.152254 2735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:00.152877 2738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:00.153654 2730 server_base.cc:1034] running on GCE node
I20240506 15:44:00.153843 2730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:00.154037 2730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:00.155170 2730 hybrid_clock.cc:648] HybridClock initialized: now 1715010240155129 us; error 96 us; skew 500 ppm
May 06 15:44:00 dist-test-slave-v0k1 krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010240, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:00.157935 2730 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:44:00.158375 2730 webserver.cc:460] Webserver started at http://127.17.240.195:41997/ using document root <none> and password file <none>
I20240506 15:44:00.158537 2730 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:00.158574 2730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:00.159909 2730 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:00.160522 2745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:00.160705 2730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:00.160784 2730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "2b18683f46a346f39dd0a44fb7278631"
format_stamp: "Formatted at 2024-05-06 15:43:18 on dist-test-slave-v0k1"
server_key: "3a08631b6289121b6109bb665eee80fb"
server_key_iv: "f237c3b285704f0af0bac07454e06e13"
server_key_version: "encryptionkey@0"
I20240506 15:44:00.161156 2730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:00.186803 2730 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:00.189043 2730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:00.189153 2730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:00.189342 2730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:00.189666 2730 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:00.189702 2730 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:00.189725 2730 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:00.189751 2730 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:00.195442 2730 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:33727
I20240506 15:44:00.195492 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:33727 every 8 connection(s)
I20240506 15:44:00.195818 2730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:44:00.196640 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 2730
I20240506 15:44:00.196861 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:00.196487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:58221)
0506 15:44:00.196659 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:00.196730 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
W20240506 15:44:00.197352 2859 heartbeater.cc:644] Failed to heartbeat to 127.17.240.254:38131 (0 consecutive failures): Network error: Failed to ping master at 127.17.240.254:38131: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
I20240506 15:44:00.197629 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:00.197535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:34721)
0506 15:44:00.197573 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:00.197606 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20240506 15:44:00.197949 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:00.197820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:44503)
0506 15:44:00.197890 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:00.197924 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20240506 15:44:00.198084 2859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240506 15:44:01.008797 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:01.008577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:48599)
0506 15:44:01.008701 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:01.008745 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20240506 15:44:01.060084 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:01.059854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:51807)
0506 15:44:01.059984 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:01.060026 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20240506 15:44:01.198709 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:01.198457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:39763)
0506 15:44:01.198597 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:01.198639 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20240506 15:44:01.725214 2866 raft_consensus.cc:482] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:01.725406 2866 raft_consensus.cc:504] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:44:01.725725 2866 leader_election.cc:288] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289), 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727)
May 06 15:44:01 dist-test-slave-v0k1 krb5kdc[1289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010239, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.195@KRBTEST.COM
May 06 15:44:01 dist-test-slave-v0k1 krb5kdc[1289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010239, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.194@KRBTEST.COM
I20240506 15:44:01.735095 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:01.726066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:33727 (local address 127.17.240.193:54359)
0506 15:44:01.726147 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:01.726157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:01.726216 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:01.726946 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:01.726950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:01.727432 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:01.727701 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:01.727706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:01.728633 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:01.728634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:01.728976 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:01.728978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:01.729088 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:01.729819 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:01.729823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:01.731482 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:01.733810 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:01.733811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:01.733811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:01.734062 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:01.734258 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:01.734259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:01.734260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:01.734316 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:01.734626 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:01.734630 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:01.734730 (+ 100us) client_negotiation.cc:769] Sending connection context
0506 15:44:01.734826 (+ 96us) client_negotiation.cc:240] Negotiation successful
0506 15:44:01.734969 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":36,"threads_started":1}
I20240506 15:44:01.735227 2867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:01.725927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:44289 (local address 127.17.240.193:44917)
0506 15:44:01.726060 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:01.726074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:01.726150 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:01.726946 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:01.726949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:01.727438 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:01.727700 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:01.727706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:01.728625 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:01.728626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:01.728975 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:01.728977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:01.729091 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:01.729746 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:01.729773 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:01.731791 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:01.733778 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:01.733784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:01.733786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:01.734069 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:01.734259 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:01.734260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:01.734261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:01.734315 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:01.734627 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:01.734632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:01.734730 (+ 98us) client_negotiation.cc:769] Sending connection context
0506 15:44:01.734825 (+ 95us) client_negotiation.cc:240] Negotiation successful
0506 15:44:01.735104 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":91,"thread_start_us":55,"threads_started":1}
I20240506 15:44:01.735127 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:01.726034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:44917 (local address 127.17.240.194:44289)
0506 15:44:01.726316 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:01.726325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:01.726341 (+ 16us) server_negotiation.cc:407] Connection header received
0506 15:44:01.726736 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:01.726738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:01.726775 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:01.726920 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:01.727817 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:01.728523 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:01.729045 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:01.729274 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:01.731889 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:01.731898 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:01.731898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:01.731913 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:01.733695 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:01.734149 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:01.734151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:01.734152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:01.734201 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:01.734396 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:01.734396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:01.734399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:01.734563 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:01.734641 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:01.734882 (+ 241us) server_negotiation.cc:299] Negotiation successful
0506 15:44:01.734995 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":59,"threads_started":1}
I20240506 15:44:01.735136 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:01.726183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:54359 (local address 127.17.240.195:33727)
0506 15:44:01.726318 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:01.726325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:01.726345 (+ 20us) server_negotiation.cc:407] Connection header received
0506 15:44:01.726736 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:01.726738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:01.726775 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:01.726908 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:01.727818 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:01.728536 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:01.729057 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:01.729290 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:01.731596 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:01.731606 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:01.731607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:01.731630 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:01.733723 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:01.734140 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:01.734143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:01.734144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:01.734199 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:01.734372 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:01.734372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:01.734373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:01.734562 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:01.734641 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:01.734883 (+ 242us) server_negotiation.cc:299] Negotiation successful
0506 15:44:01.735034 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
W20240506 15:44:01.736158 2480 leader_election.cc:341] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727): Not found: Tablet not found: ae6989ee74c147a6a3d9cb2b7710713c
W20240506 15:44:01.736248 2480 leader_election.cc:341] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289): Not found: Tablet not found: ae6989ee74c147a6a3d9cb2b7710713c
I20240506 15:44:01.736286 2480 leader_election.cc:302] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 590edb5b06bf4d2095bf5a1899592fb7; no voters: 2b18683f46a346f39dd0a44fb7278631, 2b66e0f2a22943fdbbe44b74832f622a
I20240506 15:44:01.736418 2866 raft_consensus.cc:2726] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:44:02.009435 2867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:02.009292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:52635)
0506 15:44:02.009351 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:02.009387 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20240506 15:44:02.060745 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:02.060554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:54969)
0506 15:44:02.060668 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:02.060704 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20240506 15:44:02.199483 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:02.199228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:39417)
0506 15:44:02.199377 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:02.199430 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20240506 15:44:03.010241 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:03.010019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:58911)
0506 15:44:03.010158 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:03.010188 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20240506 15:44:03.061462 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:03.061229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:33367)
0506 15:44:03.061365 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:03.061404 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20240506 15:44:03.200172 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:03.199947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:59419)
0506 15:44:03.200080 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:03.200119 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20240506 15:44:03.644604 2876 raft_consensus.cc:482] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:03.644703 2876 raft_consensus.cc:504] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:44:03.644866 2876 leader_election.cc:288] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289), 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727)
W20240506 15:44:03.645306 2480 leader_election.cc:341] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289): Not found: Tablet not found: ae6989ee74c147a6a3d9cb2b7710713c
W20240506 15:44:03.645402 2480 leader_election.cc:341] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727): Not found: Tablet not found: ae6989ee74c147a6a3d9cb2b7710713c
I20240506 15:44:03.645423 2480 leader_election.cc:302] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 590edb5b06bf4d2095bf5a1899592fb7; no voters: 2b18683f46a346f39dd0a44fb7278631, 2b66e0f2a22943fdbbe44b74832f622a
I20240506 15:44:03.645511 2876 raft_consensus.cc:2726] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:44:04.010991 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:04.010750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:45895)
0506 15:44:04.010896 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:04.010938 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20240506 15:44:04.062188 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:04.061952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:59357)
0506 15:44:04.062098 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:04.062139 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20240506 15:44:04.201045 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:04.200808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:48831)
0506 15:44:04.200941 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:04.200994 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20240506 15:44:05.011754 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:05.011527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:39229)
0506 15:44:05.011664 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:05.011703 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20240506 15:44:05.063105 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:05.062794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:44905)
0506 15:44:05.062954 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:05.062998 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20240506 15:44:05.201787 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:05.201558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:40977)
0506 15:44:05.201701 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:05.201740 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20240506 15:44:06.016763 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:06.016465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:44513)
0506 15:44:06.016643 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:06.016699 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20240506 15:44:06.063772 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:06.063568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:40905)
0506 15:44:06.063690 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:06.063722 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20240506 15:44:06.202577 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:06.202321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:34931)
0506 15:44:06.202477 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:06.202527 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20240506 15:44:06.605031 2886 raft_consensus.cc:482] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:06.605130 2886 raft_consensus.cc:504] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:44:06.605297 2886 leader_election.cc:288] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289), 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727)
W20240506 15:44:06.605690 2480 leader_election.cc:341] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727): Not found: Tablet not found: ae6989ee74c147a6a3d9cb2b7710713c
W20240506 15:44:06.605770 2480 leader_election.cc:341] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289): Not found: Tablet not found: ae6989ee74c147a6a3d9cb2b7710713c
I20240506 15:44:06.605795 2480 leader_election.cc:302] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 590edb5b06bf4d2095bf5a1899592fb7; no voters: 2b18683f46a346f39dd0a44fb7278631, 2b66e0f2a22943fdbbe44b74832f622a
I20240506 15:44:06.605887 2886 raft_consensus.cc:2726] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:44:07.017472 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:07.017231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:38215)
0506 15:44:07.017364 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:07.017402 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20240506 15:44:07.064546 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:07.064274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:50511)
0506 15:44:07.064460 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:07.064495 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":127,"threads_started":1}
I20240506 15:44:07.203306 2889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:07.203100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:54961)
0506 15:44:07.203220 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:07.203257 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20240506 15:44:08.018235 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:08.017994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:59509)
0506 15:44:08.018145 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:08.018183 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20240506 15:44:08.065222 2891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:08.065010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:60339)
0506 15:44:08.065142 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:08.065174 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20240506 15:44:08.204025 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:08.203784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:34055)
0506 15:44:08.203924 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:08.203961 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20240506 15:44:08.627207 2893 raft_consensus.cc:482] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:08.627317 2893 raft_consensus.cc:504] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:44:08.627501 2893 leader_election.cc:288] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289), 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727)
W20240506 15:44:08.628011 2480 leader_election.cc:341] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727): Not found: Tablet not found: ae6989ee74c147a6a3d9cb2b7710713c
W20240506 15:44:08.628082 2480 leader_election.cc:341] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289): Not found: Tablet not found: ae6989ee74c147a6a3d9cb2b7710713c
I20240506 15:44:08.628098 2480 leader_election.cc:302] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 590edb5b06bf4d2095bf5a1899592fb7; no voters: 2b18683f46a346f39dd0a44fb7278631, 2b66e0f2a22943fdbbe44b74832f622a
I20240506 15:44:08.628186 2893 raft_consensus.cc:2726] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:44:09.019163 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:09.018751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:39401)
0506 15:44:09.019055 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:09.019107 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":84,"threads_started":1}
I20240506 15:44:09.065819 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:09.065607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:41033)
0506 15:44:09.065733 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:09.065770 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20240506 15:44:09.204766 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:09.204534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:48361)
0506 15:44:09.204675 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:09.204714 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20240506 15:44:10.019897 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:10.019677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:53337)
0506 15:44:10.019803 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:10.019848 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20240506 15:44:10.066457 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:10.066234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:39285)
0506 15:44:10.066377 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:10.066409 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20240506 15:44:10.205518 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:10.205244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:33291)
0506 15:44:10.205426 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:10.205460 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
I20240506 15:44:11.020699 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:11.020426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:41059)
0506 15:44:11.020603 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:11.020646 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20240506 15:44:11.067132 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:11.066910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:33905)
0506 15:44:11.067043 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:11.067078 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20240506 15:44:11.206231 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:11.205988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:33703)
0506 15:44:11.206143 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:11.206180 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20240506 15:44:12.021395 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:12.021158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:36163)
0506 15:44:12.021303 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:12.021345 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20240506 15:44:12.067870 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:12.067618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:60589)
0506 15:44:12.067761 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:12.067805 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20240506 15:44:12.206853 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:12.206646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:47107)
0506 15:44:12.206766 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:12.206804 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20240506 15:44:13.022106 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:13.021852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:34153)
0506 15:44:13.021999 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:13.022036 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20240506 15:44:13.068631 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:13.068383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:48673)
0506 15:44:13.068536 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:13.068581 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20240506 15:44:13.207597 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:13.207342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:39037)
0506 15:44:13.207504 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:13.207547 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20240506 15:44:13.367424 2909 raft_consensus.cc:482] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:13.367538 2909 raft_consensus.cc:504] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b66e0f2a22943fdbbe44b74832f622a" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 44289 } } peers { permanent_uuid: "2b18683f46a346f39dd0a44fb7278631" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33727 } } peers { permanent_uuid: "590edb5b06bf4d2095bf5a1899592fb7" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37411 } }
I20240506 15:44:13.367678 2909 leader_election.cc:288] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289), 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727)
W20240506 15:44:13.368063 2480 leader_election.cc:341] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b66e0f2a22943fdbbe44b74832f622a (127.17.240.194:44289): Not found: Tablet not found: ae6989ee74c147a6a3d9cb2b7710713c
W20240506 15:44:13.368144 2480 leader_election.cc:341] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b18683f46a346f39dd0a44fb7278631 (127.17.240.195:33727): Not found: Tablet not found: ae6989ee74c147a6a3d9cb2b7710713c
I20240506 15:44:13.368177 2480 leader_election.cc:302] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 590edb5b06bf4d2095bf5a1899592fb7; no voters: 2b18683f46a346f39dd0a44fb7278631, 2b66e0f2a22943fdbbe44b74832f622a
I20240506 15:44:13.368270 2909 raft_consensus.cc:2726] T ae6989ee74c147a6a3d9cb2b7710713c P 590edb5b06bf4d2095bf5a1899592fb7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:44:14.022856 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:14.022603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:49341)
0506 15:44:14.022764 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:14.022804 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20240506 15:44:14.069305 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:14.069082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:33021)
0506 15:44:14.069213 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:14.069255 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20240506 15:44:14.208362 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:14.208113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:37263)
0506 15:44:14.208265 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:14.208308 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20240506 15:44:15.023603 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:15.023377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.193:45739)
0506 15:44:15.023522 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:15.023553 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20240506 15:44:15.070065 2914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:15.069840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.194:39229)
0506 15:44:15.069977 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:15.070014 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":97,"threads_started":1}
I20240506 15:44:15.201130 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 2429
I20240506 15:44:15.206140 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 2596
I20240506 15:44:15.209138 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:15.208945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38131 (local address 127.17.240.195:55767)
0506 15:44:15.209063 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:15.209094 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.254:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
I20240506 15:44:15.210635 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 2730
2024-05-06T15:44:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58243 ms)
[----------] 30 tests from SecurityITest (276452 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:15 dist-test-slave-v0k1 krb5kdc[2918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:15 dist-test-slave-v0k1 krb5kdc[2918](info): set up 2 sockets
May 06 15:44:15 dist-test-slave-v0k1 krb5kdc[2918](info): commencing operation
krb5kdc: starting...
W20240506 15:44:16.158865 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:16 dist-test-slave-v0k1 krb5kdc[2918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:16.283396 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39113
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:39113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:16.361927 2931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:16.362131 2931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:16.362167 2931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:16.363513 2931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:16.363561 2931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:16.363584 2931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:16.363606 2931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:16.363626 2931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:16.365273 2931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:39113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:16.365522 2931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:16.365707 2931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:16.370299 2937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:16.370968 2939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:16.371542 2936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:16.372082 2931 server_base.cc:1034] running on GCE node
I20240506 15:44:16.372285 2931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:16.372517 2931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:16.373661 2931 hybrid_clock.cc:648] HybridClock initialized: now 1715010256373634 us; error 44 us; skew 500 ppm
May 06 15:44:16 dist-test-slave-v0k1 krb5kdc[2918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010256, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:16.380357 2931 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:16.380786 2931 webserver.cc:460] Webserver started at http://127.17.240.254:45701/ using document root <none> and password file <none>
I20240506 15:44:16.380975 2931 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:16.381021 2931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:16.381125 2931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:16.382089 2931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "74d1889ba47b43c985f36926f90b2686"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "bf13a2d9107081d01e78f52cfa98908b"
server_key_iv: "9ef5fb2a3a9cb025c980295d16e5666d"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.382396 2931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "74d1889ba47b43c985f36926f90b2686"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "bf13a2d9107081d01e78f52cfa98908b"
server_key_iv: "9ef5fb2a3a9cb025c980295d16e5666d"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.383661 2931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:44:16.384317 2946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.384503 2931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:16.384582 2931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "74d1889ba47b43c985f36926f90b2686"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "bf13a2d9107081d01e78f52cfa98908b"
server_key_iv: "9ef5fb2a3a9cb025c980295d16e5666d"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.384645 2931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:16.400719 2931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:16.402714 2931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:16.402839 2931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:16.406550 2931 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:39113
I20240506 15:44:16.406615 2998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:39113 every 8 connection(s)
I20240506 15:44:16.406905 2931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:16.407626 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 2931
I20240506 15:44:16.407714 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:16.407769 2999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:16.407891 18371 external_mini_cluster.cc:1428] Setting key 953988f33a5aabfa3452df06d0b2baa1
I20240506 15:44:16.410887 2999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686: Bootstrap starting.
I20240506 15:44:16.411540 2999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:16.411809 2999 log.cc:824] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:16.412528 2999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686: No bootstrap required, opened a new log
May 06 15:44:16 dist-test-slave-v0k1 krb5kdc[2918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:16.414134 2999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d1889ba47b43c985f36926f90b2686" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39113 } }
I20240506 15:44:16.414249 2999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:16.414278 2999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d1889ba47b43c985f36926f90b2686, State: Initialized, Role: FOLLOWER
I20240506 15:44:16.414376 2999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d1889ba47b43c985f36926f90b2686" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39113 } }
I20240506 15:44:16.414448 2999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:16.414484 2999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:16.414525 2999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:16.415169 2999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d1889ba47b43c985f36926f90b2686" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39113 } }
I20240506 15:44:16.415284 2999 leader_election.cc:302] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74d1889ba47b43c985f36926f90b2686; no voters:
I20240506 15:44:16.415432 2999 leader_election.cc:288] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:16.415484 3004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:16.415663 3004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [term 1 LEADER]: Becoming Leader. State: Replica: 74d1889ba47b43c985f36926f90b2686, State: Running, Role: LEADER
I20240506 15:44:16.415668 2999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:16.415835 3004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d1889ba47b43c985f36926f90b2686" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39113 } }
I20240506 15:44:16.416220 3006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74d1889ba47b43c985f36926f90b2686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d1889ba47b43c985f36926f90b2686" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39113 } } }
I20240506 15:44:16.416363 3006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:16.416564 3006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74d1889ba47b43c985f36926f90b2686. Latest consensus state: current_term: 1 leader_uuid: "74d1889ba47b43c985f36926f90b2686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d1889ba47b43c985f36926f90b2686" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39113 } } }
I20240506 15:44:16.416677 3006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:16.416939 3013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:16.417361 3013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:16.418026 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.408398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.17.240.254:39113)
0506 15:44:16.408607 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:16.408619 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:16.408642 (+ 23us) server_negotiation.cc:407] Connection header received
0506 15:44:16.409240 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:16.409250 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:16.409305 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:16.409467 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:16.410273 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.411426 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.412009 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.412270 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.414213 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:16.414220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:16.414220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:16.414234 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:16.416396 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:16.416783 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:16.416786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:16.416786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:16.416853 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:16.417138 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:16.417139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:16.417140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:16.417376 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:16.417556 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:16.417678 (+ 122us) server_negotiation.cc:299] Negotiation successful
0506 15:44:16.417802 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":112,"threads_started":1}
I20240506 15:44:16.418991 3013 catalog_manager.cc:1347] Generated new cluster ID: bc4cde73d0f246a2822a6efa1a718e6c
I20240506 15:44:16.419070 3013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:16.440274 3013 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:16.440855 3013 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:16.446527 3013 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686: Generated new TSK 0
I20240506 15:44:16.446707 3013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:44:16.474464 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39113
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:16.557919 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:16.558118 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:16.558144 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:16.559569 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:16.559617 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:16.559643 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:44:16.561233 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:16.561447 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:16.561646 3025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:16.566593 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:16.567095 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:16.567863 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:16.568531 3025 server_base.cc:1034] running on GCE node
I20240506 15:44:16.568733 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:16.568980 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:16.570123 3025 hybrid_clock.cc:648] HybridClock initialized: now 1715010256570113 us; error 39 us; skew 500 ppm
May 06 15:44:16 dist-test-slave-v0k1 krb5kdc[2918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010256, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:16.572683 3025 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:44:16.573094 3025 webserver.cc:460] Webserver started at http://127.17.240.193:35843/ using document root <none> and password file <none>
I20240506 15:44:16.573276 3025 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:16.573323 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:16.573447 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:16.574450 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "0938862ed8ad428c9a3116bcadd2edf8"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "3d7b656db47b22c22b70d809fb8d7328"
server_key_iv: "9462c41e877f8d1bc8a2924a2b55e73b"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.574771 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "0938862ed8ad428c9a3116bcadd2edf8"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "3d7b656db47b22c22b70d809fb8d7328"
server_key_iv: "9462c41e877f8d1bc8a2924a2b55e73b"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.576071 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:44:16.576822 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.577013 3025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:16.577095 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "0938862ed8ad428c9a3116bcadd2edf8"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "3d7b656db47b22c22b70d809fb8d7328"
server_key_iv: "9462c41e877f8d1bc8a2924a2b55e73b"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.577159 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:16.597268 3025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:16.599112 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:16.599244 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:16.599471 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:16.599786 3025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:16.599819 3025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.599855 3025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:16.599889 3025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.605319 3025 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:37281
I20240506 15:44:16.605396 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:37281 every 8 connection(s)
I20240506 15:44:16.605686 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:44:16.609401 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 3025
I20240506 15:44:16.609539 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:44:16.609722 18371 external_mini_cluster.cc:1428] Setting key 17514f479e5108e8015af223d1a75902
May 06 15:44:16 dist-test-slave-v0k1 krb5kdc[2918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010256, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:16.614512 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.606365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39113 (local address 127.17.240.193:47277)
0506 15:44:16.606512 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:16.606538 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:16.606992 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:16.607227 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:16.607231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:16.607542 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:16.608146 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.608150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.608810 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.608810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:16.609205 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.609206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.609310 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.609930 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:16.609957 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:16.611401 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:16.613327 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:16.613333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:16.613336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:16.613547 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:16.613724 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:16.613725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:16.613726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:16.613802 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:16.614069 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:16.614071 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:16.614175 (+ 104us) client_negotiation.cc:769] Sending connection context
0506 15:44:16.614259 (+ 84us) client_negotiation.cc:240] Negotiation successful
0506 15:44:16.614368 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20240506 15:44:16.614532 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.606429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:47277 (local address 127.17.240.254:39113)
0506 15:44:16.606519 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:16.606522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:16.606615 (+ 93us) server_negotiation.cc:407] Connection header received
0506 15:44:16.607119 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:16.607120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:16.607130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:16.607183 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:16.608235 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.608736 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.609294 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.609491 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.611495 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:16.611501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:16.611501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:16.611509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:16.613230 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:16.613621 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:16.613623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:16.613623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:16.613654 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:16.613867 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:16.613868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:16.613869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:16.614000 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:16.614080 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:16.614299 (+ 219us) server_negotiation.cc:299] Negotiation successful
0506 15:44:16.614402 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20240506 15:44:16.615321 3154 heartbeater.cc:347] Connected to a master server at 127.17.240.254:39113
I20240506 15:44:16.615432 3154 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:16.615623 3154 heartbeater.cc:511] Master 127.17.240.254:39113 requested a full tablet report, sending...
I20240506 15:44:16.616206 2963 ts_manager.cc:194] Registered new tserver with Master: 0938862ed8ad428c9a3116bcadd2edf8 (127.17.240.193:37281)
I20240506 15:44:16.617291 2963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:47277
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:44:16.636080 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39113
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:16.715137 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:16.715317 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:16.715340 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:16.716763 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:16.716838 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:16.716876 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:44:16.718518 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:16.718719 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:16.718905 3159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:16.723587 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:16.723990 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:16.724763 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:16.725453 3159 server_base.cc:1034] running on GCE node
I20240506 15:44:16.725647 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:16.725855 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:16.726998 3159 hybrid_clock.cc:648] HybridClock initialized: now 1715010256726977 us; error 36 us; skew 500 ppm
May 06 15:44:16 dist-test-slave-v0k1 krb5kdc[2918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010256, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:16.729324 3159 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:44:16.729746 3159 webserver.cc:460] Webserver started at http://127.17.240.194:43595/ using document root <none> and password file <none>
I20240506 15:44:16.729926 3159 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:16.729964 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:16.730046 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:16.731081 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "87288a948cb448c0b6ad727bf92d7070"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "58fa163f53d30ece7ec6c46cf51d3af6"
server_key_iv: "4a6b893c8a702931d6f2eec8bace2ce9"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.731380 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "87288a948cb448c0b6ad727bf92d7070"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "58fa163f53d30ece7ec6c46cf51d3af6"
server_key_iv: "4a6b893c8a702931d6f2eec8bace2ce9"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.732491 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:44:16.733104 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.733333 3159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:16.733418 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "87288a948cb448c0b6ad727bf92d7070"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "58fa163f53d30ece7ec6c46cf51d3af6"
server_key_iv: "4a6b893c8a702931d6f2eec8bace2ce9"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.733494 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:16.742059 3159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:16.743724 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:16.743824 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:16.744030 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:16.744324 3159 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:16.744356 3159 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.744388 3159 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:16.744410 3159 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.749912 3159 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:35255
I20240506 15:44:16.749963 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:35255 every 8 connection(s)
I20240506 15:44:16.750252 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:44:16.750550 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 3159
I20240506 15:44:16.750654 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:44:16.750854 18371 external_mini_cluster.cc:1428] Setting key 72d03c1579f924e454ecee46df3710dc
May 06 15:44:16 dist-test-slave-v0k1 krb5kdc[2918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010256, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
I20240506 15:44:16.759159 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.750981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:35449 (local address 127.17.240.254:39113)
0506 15:44:16.751068 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:16.751071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:16.751260 (+ 189us) server_negotiation.cc:407] Connection header received
0506 15:44:16.751794 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:16.751795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:16.751807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:16.751867 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:16.752887 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.753447 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.753993 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.754207 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.756250 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:16.756254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:16.756255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:16.756262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:16.757854 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:16.758256 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:16.758258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:16.758258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:16.758288 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:16.758476 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:16.758476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:16.758477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:16.758592 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:16.758660 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:16.758950 (+ 290us) server_negotiation.cc:299] Negotiation successful
0506 15:44:16.759084 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240506 15:44:16.759249 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.750912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39113 (local address 127.17.240.194:35449)
0506 15:44:16.751144 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:16.751177 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:16.751697 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:16.751928 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:16.751932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:16.752225 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:16.752762 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.752769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.753533 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.753534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:16.753913 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.753914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.754020 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.754565 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:16.754590 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:16.756160 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:16.757963 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:16.757968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:16.757970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:16.758173 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:16.758356 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:16.758357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:16.758357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:16.758418 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:16.758650 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:16.758652 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:16.758757 (+ 105us) client_negotiation.cc:769] Sending connection context
0506 15:44:16.758913 (+ 156us) client_negotiation.cc:240] Negotiation successful
0506 15:44:16.759083 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20240506 15:44:16.759752 3288 heartbeater.cc:347] Connected to a master server at 127.17.240.254:39113
I20240506 15:44:16.759847 3288 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:16.760061 3288 heartbeater.cc:511] Master 127.17.240.254:39113 requested a full tablet report, sending...
I20240506 15:44:16.760465 2963 ts_manager.cc:194] Registered new tserver with Master: 87288a948cb448c0b6ad727bf92d7070 (127.17.240.194:35255)
I20240506 15:44:16.760823 2963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:35449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:44:16.777323 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39113
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:16.855288 3293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:16.855482 3293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:16.855518 3293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:16.856870 3293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:16.856913 3293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:16.856952 3293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:44:16.858457 3293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:39113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:16.858675 3293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:16.858852 3293 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:16.863529 3301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:16.864114 3298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:16.864828 3299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:16.865379 3293 server_base.cc:1034] running on GCE node
I20240506 15:44:16.865546 3293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:16.865746 3293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:16.866883 3293 hybrid_clock.cc:648] HybridClock initialized: now 1715010256866864 us; error 34 us; skew 500 ppm
May 06 15:44:16 dist-test-slave-v0k1 krb5kdc[2918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010256, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:16.869307 3293 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:44:16.869714 3293 webserver.cc:460] Webserver started at http://127.17.240.195:40745/ using document root <none> and password file <none>
I20240506 15:44:16.869870 3293 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:16.869905 3293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:16.869987 3293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:16.870918 3293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "18e33ae3795a415cb040e4540aeeaa48"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "b065a329e41857b37f28ce93f081e8c1"
server_key_iv: "ecf4079b0e2207a024ae45b238496e90"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.871251 3293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "18e33ae3795a415cb040e4540aeeaa48"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "b065a329e41857b37f28ce93f081e8c1"
server_key_iv: "ecf4079b0e2207a024ae45b238496e90"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.872422 3293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:16.873029 3308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.873194 3293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.873260 3293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "18e33ae3795a415cb040e4540aeeaa48"
format_stamp: "Formatted at 2024-05-06 15:44:16 on dist-test-slave-v0k1"
server_key: "b065a329e41857b37f28ce93f081e8c1"
server_key_iv: "ecf4079b0e2207a024ae45b238496e90"
server_key_version: "encryptionkey@0"
I20240506 15:44:16.873309 3293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:16.893581 3293 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:16.895323 3293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:16.895423 3293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:16.895596 3293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:16.895895 3293 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:16.895936 3293 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.895958 3293 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:16.895975 3293 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:16.901274 3293 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:34781
I20240506 15:44:16.901326 3421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:34781 every 8 connection(s)
I20240506 15:44:16.901612 3293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:44:16.901921 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 3293
I20240506 15:44:16.902038 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:44:16.902235 18371 external_mini_cluster.cc:1428] Setting key 9a4f8903ce327d995502e4b9daabc2eb
May 06 15:44:16 dist-test-slave-v0k1 krb5kdc[2918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010256, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:16.909811 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.902376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:32951 (local address 127.17.240.254:39113)
0506 15:44:16.902442 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:16.902445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:16.902660 (+ 215us) server_negotiation.cc:407] Connection header received
0506 15:44:16.903259 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:16.903260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:16.903269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:16.903320 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:16.904315 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.904822 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.905343 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.905447 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.907441 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:16.907447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:16.907447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:16.907454 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:16.908663 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:16.909038 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:16.909040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:16.909040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:16.909063 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:16.909225 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:16.909226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:16.909227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:16.909320 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:16.909377 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:16.909623 (+ 246us) server_negotiation.cc:299] Negotiation successful
0506 15:44:16.909715 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:44:16.909920 3424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.902355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:39113 (local address 127.17.240.195:32951)
0506 15:44:16.902532 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:16.902565 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:16.903165 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:16.903366 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:16.903369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:16.903675 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:16.904232 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.904237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.904897 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.904898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:16.905285 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.905286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.905385 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.905909 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:16.905924 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:16.907353 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:16.908748 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:16.908753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:16.908756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:16.908970 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:16.909115 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:16.909115 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:16.909116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:16.909171 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:16.909372 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:16.909377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:16.909499 (+ 122us) client_negotiation.cc:769] Sending connection context
0506 15:44:16.909597 (+ 98us) client_negotiation.cc:240] Negotiation successful
0506 15:44:16.909734 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20240506 15:44:16.910429 3422 heartbeater.cc:347] Connected to a master server at 127.17.240.254:39113
I20240506 15:44:16.910540 3422 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:16.910733 3422 heartbeater.cc:511] Master 127.17.240.254:39113 requested a full tablet report, sending...
I20240506 15:44:16.911216 2963 ts_manager.cc:194] Registered new tserver with Master: 18e33ae3795a415cb040e4540aeeaa48 (127.17.240.195:34781)
I20240506 15:44:16.911590 2963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:32951
I20240506 15:44:16.913821 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:44:16.920538 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.915106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.17.240.254:39113)
0506 15:44:16.915165 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:16.915167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:16.915307 (+ 140us) server_negotiation.cc:407] Connection header received
0506 15:44:16.915341 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:16.915342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:16.915350 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:16.915397 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:16.916033 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.916547 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.917188 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.917305 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.918396 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:16.918399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:16.918399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:16.918405 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:16.919733 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:16.919976 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:16.919977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:16.919977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:16.919992 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:16.920119 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:16.920119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:16.920120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:16.920252 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:16.920337 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:16.920400 (+ 63us) server_negotiation.cc:299] Negotiation successful
0506 15:44:16.920458 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:44:16.922072 2963 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:44:16.923533 2963 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:44:16.928365 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.925398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:34781 (local address 127.0.0.1:38934)
0506 15:44:16.925788 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:16.925805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:16.925875 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:16.926306 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:16.926307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:16.926309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:16.926519 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.926521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.927494 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.927495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:16.928080 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.928081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.928150 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.928192 (+ 42us) client_negotiation.cc:769] Sending connection context
0506 15:44:16.928246 (+ 54us) client_negotiation.cc:240] Negotiation successful
0506 15:44:16.928288 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":3,"thread_start_us":54,"threads_started":1}
I20240506 15:44:16.928678 3432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.925398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:35255 (local address 127.0.0.1:46206)
0506 15:44:16.925776 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:16.925805 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:16.925875 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:16.926069 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:16.926072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:16.926074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:16.926258 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.926259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.927546 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.927546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:16.928476 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.928478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.928541 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.928545 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:44:16.928578 (+ 33us) client_negotiation.cc:240] Negotiation successful
0506 15:44:16.928616 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":75,"threads_started":1}
I20240506 15:44:16.928917 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.925636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.17.240.195:34781)
0506 15:44:16.926009 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:16.926024 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:16.926034 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:44:16.926070 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:16.926071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:16.926087 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:16.926166 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:16.926661 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.927411 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.928172 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.928720 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.928771 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:16.928803 (+ 32us) server_negotiation.cc:299] Negotiation successful
0506 15:44:16.928845 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":72,"threads_started":1}
I20240506 15:44:16.929502 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.925546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.17.240.194:35255)
0506 15:44:16.925748 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:16.925756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:16.925885 (+ 129us) server_negotiation.cc:407] Connection header received
0506 15:44:16.925957 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:16.925959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:16.925973 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:16.926051 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:16.926328 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.927454 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.928837 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.929298 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.929338 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:16.929371 (+ 33us) server_negotiation.cc:299] Negotiation successful
0506 15:44:16.929413 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":102,"threads_started":1}
I20240506 15:44:16.929507 3356 tablet_service.cc:1467] Processing CreateTablet for tablet 967449a295d744689b591613016ec069 (DEFAULT_TABLE table=test-table [id=7928058954d744c28c912cd925a0dbcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:16.929423 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.925426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37281 (local address 127.0.0.1:39238)
0506 15:44:16.926258 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:16.926268 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:16.926320 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:16.926523 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:16.926524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:16.926526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:16.926712 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.926713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.928055 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.928055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:16.929260 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:16.929263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.929324 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.929329 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:44:16.929353 (+ 24us) client_negotiation.cc:240] Negotiation successful
0506 15:44:16.929381 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":76,"threads_started":1}
I20240506 15:44:16.929843 3356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 967449a295d744689b591613016ec069. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:16.930215 3434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:16.925637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.17.240.193:37281)
0506 15:44:16.926023 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:16.926031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:16.926375 (+ 344us) server_negotiation.cc:407] Connection header received
0506 15:44:16.926419 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:16.926421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:16.926437 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:16.926518 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:16.926797 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.927968 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:16.929498 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:16.930021 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:16.930061 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:16.930092 (+ 31us) server_negotiation.cc:299] Negotiation successful
0506 15:44:16.930139 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":70,"threads_started":1}
I20240506 15:44:16.930132 3222 tablet_service.cc:1467] Processing CreateTablet for tablet 967449a295d744689b591613016ec069 (DEFAULT_TABLE table=test-table [id=7928058954d744c28c912cd925a0dbcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:16.930418 3222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 967449a295d744689b591613016ec069. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:16.930758 3088 tablet_service.cc:1467] Processing CreateTablet for tablet 967449a295d744689b591613016ec069 (DEFAULT_TABLE table=test-table [id=7928058954d744c28c912cd925a0dbcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:16.931061 3088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 967449a295d744689b591613016ec069. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:16.932837 3436 tablet_bootstrap.cc:492] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: Bootstrap starting.
I20240506 15:44:16.933429 3436 tablet_bootstrap.cc:654] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:16.933494 3437 tablet_bootstrap.cc:492] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: Bootstrap starting.
I20240506 15:44:16.933751 3436 log.cc:824] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:16.933908 3438 tablet_bootstrap.cc:492] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8: Bootstrap starting.
I20240506 15:44:16.934041 3437 tablet_bootstrap.cc:654] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:16.934340 3437 log.cc:824] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:16.934401 3438 tablet_bootstrap.cc:654] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:16.934675 3438 log.cc:824] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:16.934689 3436 tablet_bootstrap.cc:492] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: No bootstrap required, opened a new log
I20240506 15:44:16.934751 3436 ts_tablet_manager.cc:1367] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:16.934980 3437 tablet_bootstrap.cc:492] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: No bootstrap required, opened a new log
I20240506 15:44:16.935060 3437 ts_tablet_manager.cc:1367] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:16.935495 3438 tablet_bootstrap.cc:492] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8: No bootstrap required, opened a new log
I20240506 15:44:16.935562 3438 ts_tablet_manager.cc:1367] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:16.936975 3437 raft_consensus.cc:348] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 } } peers { permanent_uuid: "87288a948cb448c0b6ad727bf92d7070" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35255 } } peers { permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 } }
I20240506 15:44:16.937036 3436 raft_consensus.cc:348] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 } } peers { permanent_uuid: "87288a948cb448c0b6ad727bf92d7070" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35255 } } peers { permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 } }
I20240506 15:44:16.937134 3437 raft_consensus.cc:374] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:16.937166 3437 raft_consensus.cc:729] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e33ae3795a415cb040e4540aeeaa48, State: Initialized, Role: FOLLOWER
I20240506 15:44:16.937170 3436 raft_consensus.cc:374] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:16.937193 3436 raft_consensus.cc:729] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87288a948cb448c0b6ad727bf92d7070, State: Initialized, Role: FOLLOWER
I20240506 15:44:16.937253 3437 consensus_queue.cc:260] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 } } peers { permanent_uuid: "87288a948cb448c0b6ad727bf92d7070" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35255 } } peers { permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 } }
I20240506 15:44:16.937270 3436 consensus_queue.cc:260] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 } } peers { permanent_uuid: "87288a948cb448c0b6ad727bf92d7070" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35255 } } peers { permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 } }
I20240506 15:44:16.937511 3436 ts_tablet_manager.cc:1398] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:16.937511 3422 heartbeater.cc:503] Master 127.17.240.254:39113 was elected leader, sending a full tablet report...
I20240506 15:44:16.937721 3288 heartbeater.cc:503] Master 127.17.240.254:39113 was elected leader, sending a full tablet report...
I20240506 15:44:16.937705 3438 raft_consensus.cc:348] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 } } peers { permanent_uuid: "87288a948cb448c0b6ad727bf92d7070" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35255 } } peers { permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 } }
I20240506 15:44:16.937819 3438 raft_consensus.cc:374] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:16.937840 3438 raft_consensus.cc:729] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0938862ed8ad428c9a3116bcadd2edf8, State: Initialized, Role: FOLLOWER
I20240506 15:44:16.937916 3438 consensus_queue.cc:260] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 } } peers { permanent_uuid: "87288a948cb448c0b6ad727bf92d7070" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35255 } } peers { permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 } }
I20240506 15:44:16.938139 3438 ts_tablet_manager.cc:1398] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:16.938182 3154 heartbeater.cc:503] Master 127.17.240.254:39113 was elected leader, sending a full tablet report...
I20240506 15:44:16.937511 3437 ts_tablet_manager.cc:1398] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240506 15:44:17.000780 3289 tablet.cc:2369] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:44:17.106360 3155 tablet.cc:2369] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 15:44:17.152137 3423 tablet.cc:2369] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:44:17.260671 3443 raft_consensus.cc:482] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:17.260795 3443 raft_consensus.cc:504] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 } } peers { permanent_uuid: "87288a948cb448c0b6ad727bf92d7070" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35255 } } peers { permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 } }
I20240506 15:44:17.261135 3443 leader_election.cc:288] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18e33ae3795a415cb040e4540aeeaa48 (127.17.240.195:34781), 0938862ed8ad428c9a3116bcadd2edf8 (127.17.240.193:37281)
I20240506 15:44:17.264305 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.261390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:34781 (local address 127.17.240.194:51323)
0506 15:44:17.261528 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:17.261550 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:17.261622 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:17.261775 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:17.261776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:17.261778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:17.262071 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.262072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.263055 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.263056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:17.264031 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.264032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.264104 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.264108 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:44:17.264144 (+ 36us) client_negotiation.cc:240] Negotiation successful
0506 15:44:17.264182 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20240506 15:44:17.264305 3445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.261344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37281 (local address 127.17.240.194:48353)
0506 15:44:17.261528 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:17.261550 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:17.261622 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:17.261775 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:17.261776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:17.261779 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:17.262070 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.262072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.263043 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.263044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:17.264021 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.264024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.264104 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.264108 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:44:17.264144 (+ 36us) client_negotiation.cc:240] Negotiation successful
0506 15:44:17.264182 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20240506 15:44:17.264717 3434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.261457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:48353 (local address 127.17.240.193:37281)
0506 15:44:17.261539 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.261542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.261606 (+ 64us) server_negotiation.cc:407] Connection header received
0506 15:44:17.261682 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.261683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.261693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.261757 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:17.262146 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.262914 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.264117 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.264599 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.264606 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.264626 (+ 20us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.264653 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240506 15:44:17.264717 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.261502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:51323 (local address 127.17.240.195:34781)
0506 15:44:17.261564 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.261566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.261631 (+ 65us) server_negotiation.cc:407] Connection header received
0506 15:44:17.261683 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.261684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.261693 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.261757 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:17.262146 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.262913 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.264115 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.264588 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.264596 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.264620 (+ 24us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.264651 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240506 15:44:17.265004 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967449a295d744689b591613016ec069" candidate_uuid: "87288a948cb448c0b6ad727bf92d7070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0938862ed8ad428c9a3116bcadd2edf8" is_pre_election: true
I20240506 15:44:17.265004 3376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967449a295d744689b591613016ec069" candidate_uuid: "87288a948cb448c0b6ad727bf92d7070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e33ae3795a415cb040e4540aeeaa48" is_pre_election: true
I20240506 15:44:17.265141 3376 raft_consensus.cc:2454] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87288a948cb448c0b6ad727bf92d7070 in term 0.
I20240506 15:44:17.265146 3108 raft_consensus.cc:2454] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87288a948cb448c0b6ad727bf92d7070 in term 0.
I20240506 15:44:17.265386 3175 leader_election.cc:302] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0938862ed8ad428c9a3116bcadd2edf8, 18e33ae3795a415cb040e4540aeeaa48, 87288a948cb448c0b6ad727bf92d7070; no voters:
I20240506 15:44:17.265570 3443 raft_consensus.cc:2781] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:44:17.265635 3443 raft_consensus.cc:482] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:44:17.265661 3443 raft_consensus.cc:3037] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:17.266376 3443 raft_consensus.cc:504] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 } } peers { permanent_uuid: "87288a948cb448c0b6ad727bf92d7070" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35255 } } peers { permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 } }
I20240506 15:44:17.266522 3443 leader_election.cc:288] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [CANDIDATE]: Term 1 election: Requested vote from peers 18e33ae3795a415cb040e4540aeeaa48 (127.17.240.195:34781), 0938862ed8ad428c9a3116bcadd2edf8 (127.17.240.193:37281)
I20240506 15:44:17.266664 3376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967449a295d744689b591613016ec069" candidate_uuid: "87288a948cb448c0b6ad727bf92d7070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e33ae3795a415cb040e4540aeeaa48"
I20240506 15:44:17.266702 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967449a295d744689b591613016ec069" candidate_uuid: "87288a948cb448c0b6ad727bf92d7070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0938862ed8ad428c9a3116bcadd2edf8"
I20240506 15:44:17.266736 3376 raft_consensus.cc:3037] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:17.266767 3108 raft_consensus.cc:3037] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:17.267516 3376 raft_consensus.cc:2454] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87288a948cb448c0b6ad727bf92d7070 in term 1.
I20240506 15:44:17.267706 3175 leader_election.cc:302] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18e33ae3795a415cb040e4540aeeaa48, 87288a948cb448c0b6ad727bf92d7070; no voters:
I20240506 15:44:17.267748 3108 raft_consensus.cc:2454] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87288a948cb448c0b6ad727bf92d7070 in term 1.
I20240506 15:44:17.267800 3443 raft_consensus.cc:2781] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:17.267964 3443 raft_consensus.cc:686] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 1 LEADER]: Becoming Leader. State: Replica: 87288a948cb448c0b6ad727bf92d7070, State: Running, Role: LEADER
I20240506 15:44:17.268075 3443 consensus_queue.cc:237] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 } } peers { permanent_uuid: "87288a948cb448c0b6ad727bf92d7070" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35255 } } peers { permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 } }
I20240506 15:44:17.268886 2963 catalog_manager.cc:5484] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87288a948cb448c0b6ad727bf92d7070 (127.17.240.194). New cstate: current_term: 1 leader_uuid: "87288a948cb448c0b6ad727bf92d7070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87288a948cb448c0b6ad727bf92d7070" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 } health_report { overall_health: UNKNOWN } } }
I20240506 15:44:17.283852 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.281832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.17.240.194:35255)
0506 15:44:17.281896 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.281898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.281907 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:44:17.281974 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.281975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.281992 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.282056 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:17.282295 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.282749 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.283381 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.283513 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.283549 (+ 36us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:17.283633 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:17.283705 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.283746 (+ 41us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.283768 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:44:17.285683 3376 raft_consensus.cc:1261] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48 [term 1 FOLLOWER]: Refusing update from remote peer 87288a948cb448c0b6ad727bf92d7070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:17.285683 3108 raft_consensus.cc:1261] T 967449a295d744689b591613016ec069 P 0938862ed8ad428c9a3116bcadd2edf8 [term 1 FOLLOWER]: Refusing update from remote peer 87288a948cb448c0b6ad727bf92d7070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:17.286027 3443 consensus_queue.cc:1035] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0938862ed8ad428c9a3116bcadd2edf8" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 37281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:17.286150 3447 consensus_queue.cc:1035] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18e33ae3795a415cb040e4540aeeaa48" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:17.288425 3450 mvcc.cc:205] Tried to move back new op lower bound from 7024682013838938112 to 7024682013770702848. Current Snapshot: MvccSnapshot[applied={T|T < 7024682013838938112}]
I20240506 15:44:17.288969 3452 mvcc.cc:205] Tried to move back new op lower bound from 7024682013838938112 to 7024682013770702848. Current Snapshot: MvccSnapshot[applied={T|T < 7024682013838938112}]
I20240506 15:44:17.289389 3451 mvcc.cc:205] Tried to move back new op lower bound from 7024682013838938112 to 7024682013770702848. Current Snapshot: MvccSnapshot[applied={T|T < 7024682013838938112 or (T in {7024682013838938112})}]
I20240506 15:44:17.291160 2963 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32794:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:17.291247 2963 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32794:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:17.292030 2963 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 74d1889ba47b43c985f36926f90b2686: Sending DeleteTablet for 3 replicas of tablet 967449a295d744689b591613016ec069
I20240506 15:44:17.292351 3356 tablet_service.cc:1514] Processing DeleteTablet for tablet 967449a295d744689b591613016ec069 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:17 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:38934
I20240506 15:44:17.292361 3222 tablet_service.cc:1514] Processing DeleteTablet for tablet 967449a295d744689b591613016ec069 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:17 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:46206
I20240506 15:44:17.292577 3460 tablet_replica.cc:330] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: stopping tablet replica
I20240506 15:44:17.292708 3460 raft_consensus.cc:2229] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:44:17.292765 3088 tablet_service.cc:1514] Processing DeleteTablet for tablet 967449a295d744689b591613016ec069 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:17 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:39238
I20240506 15:44:17.292858 3460 raft_consensus.cc:2258] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:17.292923 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 3025
I20240506 15:44:17.293476 3460 ts_tablet_manager.cc:1875] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:44:17.295244 3461 tablet_replica.cc:330] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: stopping tablet replica
I20240506 15:44:17.295238 3460 ts_tablet_manager.cc:1888] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:44:17.295308 3460 log.cc:1196] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/967449a295d744689b591613016ec069
I20240506 15:44:17.295333 3461 raft_consensus.cc:2229] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:44:17.295495 3461 raft_consensus.cc:2258] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:17.295962 3461 ts_tablet_manager.cc:1875] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:44:17.297488 3461 ts_tablet_manager.cc:1888] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:44:17.297564 3461 log.cc:1196] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/967449a295d744689b591613016ec069
I20240506 15:44:17.297814 3461 ts_tablet_manager.cc:1909] T 967449a295d744689b591613016ec069 P 87288a948cb448c0b6ad727bf92d7070: Deleting consensus metadata
I20240506 15:44:17.298204 3460 ts_tablet_manager.cc:1909] T 967449a295d744689b591613016ec069 P 18e33ae3795a415cb040e4540aeeaa48: Deleting consensus metadata
I20240506 15:44:17.298214 2950 catalog_manager.cc:4830] TS 87288a948cb448c0b6ad727bf92d7070 (127.17.240.194:35255): tablet 967449a295d744689b591613016ec069 (table test-table [id=7928058954d744c28c912cd925a0dbcb]) successfully deleted
I20240506 15:44:17.298600 2947 catalog_manager.cc:4830] TS 18e33ae3795a415cb040e4540aeeaa48 (127.17.240.195:34781): tablet 967449a295d744689b591613016ec069 (table test-table [id=7928058954d744c28c912cd925a0dbcb]) successfully deleted
W20240506 15:44:17.298862 2948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:37281 (error 108)
I20240506 15:44:17.299155 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 3159
I20240506 15:44:17.299965 3432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.299310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:37281 (local address 127.0.0.1:39246)
0506 15:44:17.299895 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:17.299935 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565}
W20240506 15:44:17.300105 2948 catalog_manager.cc:4573] TS 0938862ed8ad428c9a3116bcadd2edf8 (127.17.240.193:37281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 967449a295d744689b591613016ec069: Network error: Client connection negotiation failed: client connection to 127.17.240.193:37281: connect: Connection refused (error 111)
I20240506 15:44:17.304378 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 3293
I20240506 15:44:17.309171 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 2931
2024-05-06T15:44:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2098 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2024-05-06T15:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:17Z Disabled control of system clock
I20240506 15:44:17.335561 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36993
--webserver_interface=127.17.240.254
--webserver_port=0
--builtin_ntp_servers=127.17.240.212:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240506 15:44:17.418349 3467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:17.418560 3467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:17.418582 3467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:17.419924 3467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:17.419966 3467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:17.419978 3467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:17.419991 3467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:17.420002 3467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:17.421437 3467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:36993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:17.421620 3467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:17.421824 3467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:17.426661 3473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:17.427071 3475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:17.427796 3472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:17.428531 3467 server_base.cc:1034] running on GCE node
I20240506 15:44:17.428764 3467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:17.429091 3467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:17.430223 3467 hybrid_clock.cc:648] HybridClock initialized: now 1715010257430214 us; error 31 us; skew 500 ppm
I20240506 15:44:17.431397 3467 webserver.cc:460] Webserver started at http://127.17.240.254:41961/ using document root <none> and password file <none>
I20240506 15:44:17.431584 3467 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:17.431627 3467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:17.431740 3467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:17.432719 3467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "1f7ba251d56f9f17b315967ab0242b01"
server_key_iv: "54646085c197b05cae61ca45f2d12ac5"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.433055 3467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "1f7ba251d56f9f17b315967ab0242b01"
server_key_iv: "54646085c197b05cae61ca45f2d12ac5"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.434139 3467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:44:17.434736 3481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.434877 3467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.434952 3467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "1f7ba251d56f9f17b315967ab0242b01"
server_key_iv: "54646085c197b05cae61ca45f2d12ac5"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.435046 3467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:17.450594 3467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:17.450886 3467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:17.451001 3467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:17.454975 3467 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:36993
I20240506 15:44:17.455039 3533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:36993 every 8 connection(s)
I20240506 15:44:17.455399 3467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:17.456032 3534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:17.458575 3534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5: Bootstrap starting.
I20240506 15:44:17.459161 3534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:17.459431 3534 log.cc:824] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:17.459494 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 3467
I20240506 15:44:17.459599 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:17.459793 18371 external_mini_cluster.cc:1428] Setting key 3551887bff45b53d993fbc509a0e012b
I20240506 15:44:17.460119 3534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5: No bootstrap required, opened a new log
I20240506 15:44:17.461665 3534 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36993 } }
I20240506 15:44:17.461784 3534 raft_consensus.cc:374] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:17.461813 3534 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53d4c715f374bbab9e7a3b7ebf0e2e5, State: Initialized, Role: FOLLOWER
I20240506 15:44:17.461968 3534 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36993 } }
I20240506 15:44:17.462034 3534 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:17.462066 3534 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:17.462126 3534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:17.462677 3534 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36993 } }
I20240506 15:44:17.462776 3534 leader_election.cc:302] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53d4c715f374bbab9e7a3b7ebf0e2e5; no voters:
I20240506 15:44:17.462945 3534 leader_election.cc:288] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:17.462988 3539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:17.463147 3539 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [term 1 LEADER]: Becoming Leader. State: Replica: e53d4c715f374bbab9e7a3b7ebf0e2e5, State: Running, Role: LEADER
I20240506 15:44:17.463232 3539 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36993 } }
I20240506 15:44:17.463241 3534 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:17.463539 3540 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36993 } } }
I20240506 15:44:17.463707 3540 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:17.463518 3541 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53d4c715f374bbab9e7a3b7ebf0e2e5. Latest consensus state: current_term: 1 leader_uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d4c715f374bbab9e7a3b7ebf0e2e5" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36993 } } }
I20240506 15:44:17.464103 3541 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:17.464179 3543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:17.464696 3543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:17.466256 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.460338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.17.240.254:36993)
0506 15:44:17.460647 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.460653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.460673 (+ 20us) server_negotiation.cc:407] Connection header received
0506 15:44:17.461649 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.461654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.461704 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.461857 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:17.462293 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.463732 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.464554 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.464817 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.465333 (+ 516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:17.465343 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:17.465343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:17.465536 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:17.465569 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:17.465576 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:17.465647 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.466027 (+ 380us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.466086 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
I20240506 15:44:17.466311 3543 catalog_manager.cc:1347] Generated new cluster ID: 6e1ecf026c654780b555afac2da9cc0c
I20240506 15:44:17.466356 3543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:17.482808 3543 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:17.483425 3543 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:17.491286 3543 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5: Generated new TSK 0
I20240506 15:44:17.491519 3543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:17.499567 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36993
--builtin_ntp_servers=127.17.240.212:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240506 15:44:17.579345 3558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:17.579530 3558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:17.579563 3558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:17.580865 3558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:17.580907 3558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:17.580943 3558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:44:17.582350 3558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:17.582567 3558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:17.582803 3558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:17.588320 3564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:17.588966 3566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:17.589751 3563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:17.590389 3558 server_base.cc:1034] running on GCE node
I20240506 15:44:17.590574 3558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:17.590788 3558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:17.591943 3558 hybrid_clock.cc:648] HybridClock initialized: now 1715010257591920 us; error 39 us; skew 500 ppm
I20240506 15:44:17.593268 3558 webserver.cc:460] Webserver started at http://127.17.240.193:42293/ using document root <none> and password file <none>
I20240506 15:44:17.593484 3558 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:17.593528 3558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:17.593627 3558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:17.594676 3558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "8c3ca88c73464b978dacfa6691bc6eec"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "c3b54d29404f75db6d5c515d9f81ffd1"
server_key_iv: "258f209df09fb721c15dbb0fd45fb071"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.594970 3558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "8c3ca88c73464b978dacfa6691bc6eec"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "c3b54d29404f75db6d5c515d9f81ffd1"
server_key_iv: "258f209df09fb721c15dbb0fd45fb071"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.596186 3558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:44:17.596874 3572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.597090 3558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:17.597190 3558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "8c3ca88c73464b978dacfa6691bc6eec"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "c3b54d29404f75db6d5c515d9f81ffd1"
server_key_iv: "258f209df09fb721c15dbb0fd45fb071"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.597255 3558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:17.618077 3558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:17.618397 3558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:17.618531 3558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:17.618757 3558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:17.619122 3558 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:17.619176 3558 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.619207 3558 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:17.619231 3558 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.626137 3558 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:46647
I20240506 15:44:17.626191 3685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:46647 every 8 connection(s)
I20240506 15:44:17.626521 3558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:44:17.631134 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.627321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:37273 (local address 127.17.240.254:36993)
0506 15:44:17.627370 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.627372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.627599 (+ 227us) server_negotiation.cc:407] Connection header received
0506 15:44:17.628095 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.628097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.628110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.628169 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:17.628624 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.629099 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.629624 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.629791 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.630815 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:17.630820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:17.630820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:17.630824 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:17.630836 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:17.630840 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:17.630886 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.630994 (+ 108us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.631050 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240506 15:44:17.631309 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.627303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36993 (local address 127.17.240.193:37273)
0506 15:44:17.627491 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:17.627514 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:17.627990 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:17.628233 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:17.628236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:17.628244 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:17.628537 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.628539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.629189 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.629190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:17.629543 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.629545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.629669 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.630683 (+ 1014us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:44:17.630684 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:17.630701 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:44:17.630718 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:17.630895 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:17.630899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:17.630902 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:44:17.630987 (+ 85us) client_negotiation.cc:240] Negotiation successful
0506 15:44:17.631075 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20240506 15:44:17.631774 3686 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36993
I20240506 15:44:17.631879 3686 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:17.632103 3686 heartbeater.cc:511] Master 127.17.240.254:36993 requested a full tablet report, sending...
I20240506 15:44:17.632655 3494 ts_manager.cc:194] Registered new tserver with Master: 8c3ca88c73464b978dacfa6691bc6eec (127.17.240.193:46647)
I20240506 15:44:17.633343 3494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.193:37273
I20240506 15:44:17.634799 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 3558
I20240506 15:44:17.634871 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:44:17.635073 18371 external_mini_cluster.cc:1428] Setting key e99f67036a655ff147767b77b5abd5fb
I20240506 15:44:17.636001 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36993
--builtin_ntp_servers=127.17.240.212:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240506 15:44:17.713842 3689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:17.714010 3689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:17.714031 3689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:17.715312 3689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:17.715353 3689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:17.715395 3689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:44:17.716723 3689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:17.716892 3689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:17.717078 3689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:17.721746 3695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:17.722433 3697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:17.723022 3694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:17.723654 3689 server_base.cc:1034] running on GCE node
I20240506 15:44:17.723825 3689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:17.724037 3689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:17.725167 3689 hybrid_clock.cc:648] HybridClock initialized: now 1715010257725160 us; error 33 us; skew 500 ppm
I20240506 15:44:17.726382 3689 webserver.cc:460] Webserver started at http://127.17.240.194:33343/ using document root <none> and password file <none>
I20240506 15:44:17.726585 3689 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:17.726634 3689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:17.726747 3689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:17.727747 3689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "6b8712cab0ef43ceb8a639e717a39160"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "5064f962e0c01550018b70d4e5dfbbae"
server_key_iv: "0482adb91ce53d4f32730d8b61e7bb34"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.728065 3689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "6b8712cab0ef43ceb8a639e717a39160"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "5064f962e0c01550018b70d4e5dfbbae"
server_key_iv: "0482adb91ce53d4f32730d8b61e7bb34"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.729174 3689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:44:17.729774 3703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.729926 3689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:17.730023 3689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "6b8712cab0ef43ceb8a639e717a39160"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "5064f962e0c01550018b70d4e5dfbbae"
server_key_iv: "0482adb91ce53d4f32730d8b61e7bb34"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.730080 3689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:17.745790 3689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:17.745995 3689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:17.746075 3689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:17.746214 3689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:17.746443 3689 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:17.746470 3689 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.746498 3689 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:17.746515 3689 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.751937 3689 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:35121
I20240506 15:44:17.751979 3816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:35121 every 8 connection(s)
I20240506 15:44:17.752250 3689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:44:17.756875 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.752896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:45643 (local address 127.17.240.254:36993)
0506 15:44:17.752975 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.752978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.753149 (+ 171us) server_negotiation.cc:407] Connection header received
0506 15:44:17.753648 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.753649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.753662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.753722 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:17.754176 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.754661 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.755219 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.755389 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.756560 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:17.756564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:17.756564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:17.756569 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:17.756590 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:17.756593 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:17.756643 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.756737 (+ 94us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.756784 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240506 15:44:17.757001 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.752844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36993 (local address 127.17.240.194:45643)
0506 15:44:17.753046 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:17.753071 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:17.753539 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:17.753786 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:17.753789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:17.753796 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:17.754086 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.754088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.754742 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.754743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:17.755140 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.755142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.755270 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.756255 (+ 985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:44:17.756256 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:17.756273 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:44:17.756296 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:17.756655 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:17.756661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:17.756664 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:44:17.756738 (+ 74us) client_negotiation.cc:240] Negotiation successful
0506 15:44:17.756822 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20240506 15:44:17.757520 3817 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36993
I20240506 15:44:17.757611 3817 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:17.757776 3817 heartbeater.cc:511] Master 127.17.240.254:36993 requested a full tablet report, sending...
I20240506 15:44:17.758106 3494 ts_manager.cc:194] Registered new tserver with Master: 6b8712cab0ef43ceb8a639e717a39160 (127.17.240.194:35121)
I20240506 15:44:17.758476 3494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.194:45643
I20240506 15:44:17.760509 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 3689
I20240506 15:44:17.760591 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:44:17.760739 18371 external_mini_cluster.cc:1428] Setting key 7a4ed348caea3f7a2ba15afecff59184
I20240506 15:44:17.761634 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36993
--builtin_ntp_servers=127.17.240.212:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240506 15:44:17.840308 3820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:17.840497 3820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:17.840526 3820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:17.841790 3820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:17.841835 3820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:17.841872 3820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:44:17.843714 3820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:36993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:17.843942 3820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:17.844180 3820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:17.848739 3828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:17.849359 3826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:17.849879 3825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:17.850477 3820 server_base.cc:1034] running on GCE node
I20240506 15:44:17.850617 3820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:17.850793 3820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:17.851953 3820 hybrid_clock.cc:648] HybridClock initialized: now 1715010257851938 us; error 28 us; skew 500 ppm
I20240506 15:44:17.853399 3820 webserver.cc:460] Webserver started at http://127.17.240.195:34915/ using document root <none> and password file <none>
I20240506 15:44:17.853610 3820 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:17.853664 3820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:17.853796 3820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:17.854815 3820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "ca04530d62ee4d1a9879f03e06884069"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "0385acdc36358943113539ecc0d0fd96"
server_key_iv: "ce0b662c4e5e2ef831c2b25bb0068c8c"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.855141 3820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "ca04530d62ee4d1a9879f03e06884069"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "0385acdc36358943113539ecc0d0fd96"
server_key_iv: "ce0b662c4e5e2ef831c2b25bb0068c8c"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.856433 3820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:17.857151 3834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.857353 3820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:17.857441 3820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "ca04530d62ee4d1a9879f03e06884069"
format_stamp: "Formatted at 2024-05-06 15:44:17 on dist-test-slave-v0k1"
server_key: "0385acdc36358943113539ecc0d0fd96"
server_key_iv: "ce0b662c4e5e2ef831c2b25bb0068c8c"
server_key_version: "encryptionkey@0"
I20240506 15:44:17.857506 3820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:17.883800 3820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:17.884124 3820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:17.884245 3820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:17.884457 3820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:17.884776 3820 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:17.884809 3820 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.884846 3820 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:17.884886 3820 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:17.890154 3820 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:36093
I20240506 15:44:17.890221 3947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:36093 every 8 connection(s)
I20240506 15:44:17.890498 3820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:44:17.894841 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.891200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:56809 (local address 127.17.240.254:36993)
0506 15:44:17.891264 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.891266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.891462 (+ 196us) server_negotiation.cc:407] Connection header received
0506 15:44:17.891929 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.891931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.891949 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.892004 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:17.892461 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.892931 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.893473 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.893645 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.894574 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:17.894578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:17.894579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:17.894584 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:17.894598 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:17.894601 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:17.894645 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.894730 (+ 85us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.894764 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240506 15:44:17.894977 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.891161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:36993 (local address 127.17.240.195:56809)
0506 15:44:17.891341 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:17.891365 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:17.891826 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:17.892070 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:17.892073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:17.892081 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:17.892370 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.892373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.893032 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.893033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:17.893389 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.893391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.893517 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.894454 (+ 937us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:44:17.894455 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:17.894472 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:44:17.894488 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:17.894653 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:17.894657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:17.894660 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:44:17.894728 (+ 68us) client_negotiation.cc:240] Negotiation successful
0506 15:44:17.894791 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20240506 15:44:17.895524 3948 heartbeater.cc:347] Connected to a master server at 127.17.240.254:36993
I20240506 15:44:17.895609 3948 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:17.895769 3948 heartbeater.cc:511] Master 127.17.240.254:36993 requested a full tablet report, sending...
I20240506 15:44:17.896127 3494 ts_manager.cc:194] Registered new tserver with Master: ca04530d62ee4d1a9879f03e06884069 (127.17.240.195:36093)
I20240506 15:44:17.896404 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 3820
I20240506 15:44:17.896539 3494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.195:56809
I20240506 15:44:17.896498 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:44:17.896778 18371 external_mini_cluster.cc:1428] Setting key 29af86f61c1fa3693b1f13c6eafad7bc
I20240506 15:44:17.897629 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:44:17.901348 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.899048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.17.240.254:36993)
0506 15:44:17.899086 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.899088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.899232 (+ 144us) server_negotiation.cc:407] Connection header received
0506 15:44:17.899277 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.899278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.899286 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.899331 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:17.899573 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.900018 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.900564 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.900670 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.901117 (+ 447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:17.901120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:17.901120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:17.901124 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:17.901136 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:17.901139 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:17.901188 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.901272 (+ 84us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.901296 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240506 15:44:17.902945 3494 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:44:17.904374 3494 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:44:17.909030 3956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.906330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:36093 (local address 127.0.0.1:40516)
0506 15:44:17.906459 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:17.906487 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:17.906555 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:17.906901 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:17.906904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:17.906907 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:17.907294 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.907296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.908179 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.908180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:17.908758 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.908760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.908831 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.908866 (+ 35us) client_negotiation.cc:769] Sending connection context
0506 15:44:17.908914 (+ 48us) client_negotiation.cc:240] Negotiation successful
0506 15:44:17.908958 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20240506 15:44:17.909819 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.906352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:35121 (local address 127.0.0.1:55794)
0506 15:44:17.906606 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:17.906618 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:17.906675 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:17.907165 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:17.907166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:17.907168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:17.907458 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.907461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.908765 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.908767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:17.909612 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.909614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.909691 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.909694 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:44:17.909723 (+ 29us) client_negotiation.cc:240] Negotiation successful
0506 15:44:17.909757 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":95,"threads_started":1}
I20240506 15:44:17.909837 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.906577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.17.240.195:36093)
0506 15:44:17.906721 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.906729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.906738 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:44:17.906772 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.906774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.906792 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.906882 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:17.907399 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.908105 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.908839 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.909574 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.909685 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.909716 (+ 31us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.909760 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20240506 15:44:17.910511 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.906330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:46647 (local address 127.0.0.1:54976)
0506 15:44:17.906894 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:17.906903 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:17.906968 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:17.907142 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:17.907142 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:17.907144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:17.907335 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.907337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.908713 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.908714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:17.909661 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:17.909664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.910327 (+ 663us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.910331 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:44:17.910404 (+ 73us) client_negotiation.cc:240] Negotiation successful
0506 15:44:17.910448 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":151,"thread_start_us":69,"threads_started":1}
I20240506 15:44:17.910540 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.906431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.17.240.193:46647)
0506 15:44:17.906640 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.906648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.906970 (+ 322us) server_negotiation.cc:407] Connection header received
0506 15:44:17.907052 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.907054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.907069 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.907141 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:17.907420 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.908614 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.909781 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.910252 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.910301 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.910423 (+ 122us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.910461 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20240506 15:44:17.910494 3882 tablet_service.cc:1467] Processing CreateTablet for tablet a5ed8721c03244b199918a1a24bdc5bd (DEFAULT_TABLE table=test-table [id=39159b9c2ebc46f1a35f15468ae5c86f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:17.910802 3882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ed8721c03244b199918a1a24bdc5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:17.910845 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:17.906738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.17.240.194:35121)
0506 15:44:17.906914 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:17.906926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:17.906935 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:44:17.906973 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:17.906975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:17.906990 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:17.907088 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:17.907553 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.908684 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:17.909703 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:17.910491 (+ 788us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:17.910558 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:17.910720 (+ 162us) server_negotiation.cc:299] Negotiation successful
0506 15:44:17.910763 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20240506 15:44:17.911255 3620 tablet_service.cc:1467] Processing CreateTablet for tablet a5ed8721c03244b199918a1a24bdc5bd (DEFAULT_TABLE table=test-table [id=39159b9c2ebc46f1a35f15468ae5c86f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:17.911341 3751 tablet_service.cc:1467] Processing CreateTablet for tablet a5ed8721c03244b199918a1a24bdc5bd (DEFAULT_TABLE table=test-table [id=39159b9c2ebc46f1a35f15468ae5c86f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:17.911525 3620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ed8721c03244b199918a1a24bdc5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:17.911625 3751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ed8721c03244b199918a1a24bdc5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:17.913980 3962 tablet_bootstrap.cc:492] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160: Bootstrap starting.
I20240506 15:44:17.914242 3963 tablet_bootstrap.cc:492] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: Bootstrap starting.
I20240506 15:44:17.914541 3964 tablet_bootstrap.cc:492] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec: Bootstrap starting.
I20240506 15:44:17.914904 3962 tablet_bootstrap.cc:654] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:17.914904 3963 tablet_bootstrap.cc:654] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:17.915087 3964 tablet_bootstrap.cc:654] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:17.915244 3962 log.cc:824] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:17.915244 3963 log.cc:824] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:17.915331 3964 log.cc:824] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:17.916004 3963 tablet_bootstrap.cc:492] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: No bootstrap required, opened a new log
I20240506 15:44:17.916016 3964 tablet_bootstrap.cc:492] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec: No bootstrap required, opened a new log
I20240506 15:44:17.916076 3964 ts_tablet_manager.cc:1367] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240506 15:44:17.916079 3963 ts_tablet_manager.cc:1367] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:17.916203 3962 tablet_bootstrap.cc:492] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160: No bootstrap required, opened a new log
I20240506 15:44:17.916246 3962 ts_tablet_manager.cc:1367] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:17.917685 3964 raft_consensus.cc:348] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 } } peers { permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 } } peers { permanent_uuid: "6b8712cab0ef43ceb8a639e717a39160" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35121 } }
I20240506 15:44:17.917809 3964 raft_consensus.cc:374] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:17.917831 3964 raft_consensus.cc:729] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3ca88c73464b978dacfa6691bc6eec, State: Initialized, Role: FOLLOWER
I20240506 15:44:17.917918 3964 consensus_queue.cc:260] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 } } peers { permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 } } peers { permanent_uuid: "6b8712cab0ef43ceb8a639e717a39160" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35121 } }
I20240506 15:44:17.918121 3964 ts_tablet_manager.cc:1398] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 15:44:17.918171 3686 heartbeater.cc:503] Master 127.17.240.254:36993 was elected leader, sending a full tablet report...
I20240506 15:44:17.918675 3962 raft_consensus.cc:348] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 } } peers { permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 } } peers { permanent_uuid: "6b8712cab0ef43ceb8a639e717a39160" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35121 } }
I20240506 15:44:17.918839 3962 raft_consensus.cc:374] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:17.918898 3962 raft_consensus.cc:729] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8712cab0ef43ceb8a639e717a39160, State: Initialized, Role: FOLLOWER
I20240506 15:44:17.918896 3963 raft_consensus.cc:348] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 } } peers { permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 } } peers { permanent_uuid: "6b8712cab0ef43ceb8a639e717a39160" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35121 } }
I20240506 15:44:17.919070 3963 raft_consensus.cc:374] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:17.919034 3962 consensus_queue.cc:260] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 } } peers { permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 } } peers { permanent_uuid: "6b8712cab0ef43ceb8a639e717a39160" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35121 } }
I20240506 15:44:17.919134 3963 raft_consensus.cc:729] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca04530d62ee4d1a9879f03e06884069, State: Initialized, Role: FOLLOWER
I20240506 15:44:17.919277 3962 ts_tablet_manager.cc:1398] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:17.919240 3963 consensus_queue.cc:260] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 } } peers { permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 } } peers { permanent_uuid: "6b8712cab0ef43ceb8a639e717a39160" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35121 } }
I20240506 15:44:17.919389 3817 heartbeater.cc:503] Master 127.17.240.254:36993 was elected leader, sending a full tablet report...
I20240506 15:44:17.919481 3963 ts_tablet_manager.cc:1398] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:17.919523 3948 heartbeater.cc:503] Master 127.17.240.254:36993 was elected leader, sending a full tablet report...
W20240506 15:44:18.002743 3818 tablet.cc:2369] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:44:18.024508 3969 raft_consensus.cc:482] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:18.024612 3969 raft_consensus.cc:504] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 } } peers { permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 } } peers { permanent_uuid: "6b8712cab0ef43ceb8a639e717a39160" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35121 } }
I20240506 15:44:18.024948 3969 leader_election.cc:288] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca04530d62ee4d1a9879f03e06884069 (127.17.240.195:36093), 8c3ca88c73464b978dacfa6691bc6eec (127.17.240.193:46647)
I20240506 15:44:18.026975 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:18.025097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:46647 (local address 127.17.240.194:52587)
0506 15:44:18.025149 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:18.025161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:18.025202 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:18.025343 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:18.025344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:18.025346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:18.025534 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:18.025536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:18.026217 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:18.026218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:18.026756 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:18.026757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:18.026825 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:18.026829 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:44:18.026861 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:44:18.026900 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20240506 15:44:18.027114 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:18.025098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:36093 (local address 127.17.240.194:50937)
0506 15:44:18.025288 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:18.025297 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:18.025360 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:18.025495 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:18.025496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:18.025498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:18.025699 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:18.025701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:18.026308 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:18.026309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:18.026860 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:18.026861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:18.026937 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:18.026941 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:44:18.026973 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:44:18.027037 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":8,"thread_start_us":96,"threads_started":1}
I20240506 15:44:18.027453 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:18.025172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:52587 (local address 127.17.240.193:46647)
0506 15:44:18.025228 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:18.025230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:18.025238 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:44:18.025266 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:18.025266 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:18.025277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:18.025328 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:18.025621 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:18.026120 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:18.026844 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:18.027341 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:18.027349 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:18.027372 (+ 23us) server_negotiation.cc:299] Negotiation successful
0506 15:44:18.027408 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240506 15:44:18.027572 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:18.025211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:50937 (local address 127.17.240.195:36093)
0506 15:44:18.025271 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:18.025273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:18.025356 (+ 83us) server_negotiation.cc:407] Connection header received
0506 15:44:18.025423 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:18.025424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:18.025434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:18.025490 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:18.025758 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:18.026237 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:18.026939 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:18.027446 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:18.027452 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:18.027470 (+ 18us) server_negotiation.cc:299] Negotiation successful
0506 15:44:18.027493 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240506 15:44:18.027730 3640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ed8721c03244b199918a1a24bdc5bd" candidate_uuid: "6b8712cab0ef43ceb8a639e717a39160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3ca88c73464b978dacfa6691bc6eec" is_pre_election: true
I20240506 15:44:18.027865 3640 raft_consensus.cc:2454] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8712cab0ef43ceb8a639e717a39160 in term 0.
I20240506 15:44:18.027853 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ed8721c03244b199918a1a24bdc5bd" candidate_uuid: "6b8712cab0ef43ceb8a639e717a39160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca04530d62ee4d1a9879f03e06884069" is_pre_election: true
I20240506 15:44:18.027976 3902 raft_consensus.cc:2454] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8712cab0ef43ceb8a639e717a39160 in term 0.
I20240506 15:44:18.028082 3707 leader_election.cc:302] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8712cab0ef43ceb8a639e717a39160, 8c3ca88c73464b978dacfa6691bc6eec; no voters:
I20240506 15:44:18.028254 3969 raft_consensus.cc:2781] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:44:18.028317 3969 raft_consensus.cc:482] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:44:18.028347 3969 raft_consensus.cc:3037] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:18.029008 3969 raft_consensus.cc:504] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 } } peers { permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 } } peers { permanent_uuid: "6b8712cab0ef43ceb8a639e717a39160" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35121 } }
I20240506 15:44:18.029157 3969 leader_election.cc:288] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [CANDIDATE]: Term 1 election: Requested vote from peers ca04530d62ee4d1a9879f03e06884069 (127.17.240.195:36093), 8c3ca88c73464b978dacfa6691bc6eec (127.17.240.193:46647)
I20240506 15:44:18.029299 3640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ed8721c03244b199918a1a24bdc5bd" candidate_uuid: "6b8712cab0ef43ceb8a639e717a39160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3ca88c73464b978dacfa6691bc6eec"
I20240506 15:44:18.029361 3640 raft_consensus.cc:3037] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:18.030019 3640 raft_consensus.cc:2454] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8712cab0ef43ceb8a639e717a39160 in term 1.
I20240506 15:44:18.030126 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ed8721c03244b199918a1a24bdc5bd" candidate_uuid: "6b8712cab0ef43ceb8a639e717a39160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca04530d62ee4d1a9879f03e06884069"
I20240506 15:44:18.030184 3707 leader_election.cc:302] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8712cab0ef43ceb8a639e717a39160, 8c3ca88c73464b978dacfa6691bc6eec; no voters:
I20240506 15:44:18.030201 3902 raft_consensus.cc:3037] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:18.030256 3969 raft_consensus.cc:2781] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:18.030396 3969 raft_consensus.cc:686] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 1 LEADER]: Becoming Leader. State: Replica: 6b8712cab0ef43ceb8a639e717a39160, State: Running, Role: LEADER
I20240506 15:44:18.030483 3969 consensus_queue.cc:237] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 } } peers { permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 } } peers { permanent_uuid: "6b8712cab0ef43ceb8a639e717a39160" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35121 } }
I20240506 15:44:18.031071 3902 raft_consensus.cc:2454] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8712cab0ef43ceb8a639e717a39160 in term 1.
I20240506 15:44:18.031280 3494 catalog_manager.cc:5484] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b8712cab0ef43ceb8a639e717a39160 (127.17.240.194). New cstate: current_term: 1 leader_uuid: "6b8712cab0ef43ceb8a639e717a39160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b8712cab0ef43ceb8a639e717a39160" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 35121 } health_report { overall_health: HEALTHY } } }
I20240506 15:44:18.053985 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:18.052165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.17.240.194:35121)
0506 15:44:18.052218 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:18.052220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:18.052226 (+ 6us) server_negotiation.cc:407] Connection header received
0506 15:44:18.052264 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:18.052264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:18.052281 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:18.052344 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:18.052547 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:18.053033 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:18.053530 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:18.053639 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:18.053685 (+ 46us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:18.053778 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:18.053837 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:18.053868 (+ 31us) server_negotiation.cc:299] Negotiation successful
0506 15:44:18.053885 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240506 15:44:18.055655 3902 raft_consensus.cc:1261] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069 [term 1 FOLLOWER]: Refusing update from remote peer 6b8712cab0ef43ceb8a639e717a39160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:18.055655 3640 raft_consensus.cc:1261] T a5ed8721c03244b199918a1a24bdc5bd P 8c3ca88c73464b978dacfa6691bc6eec [term 1 FOLLOWER]: Refusing update from remote peer 6b8712cab0ef43ceb8a639e717a39160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:18.056002 3972 consensus_queue.cc:1035] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3ca88c73464b978dacfa6691bc6eec" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:18.056202 3969 consensus_queue.cc:1035] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca04530d62ee4d1a9879f03e06884069" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:18.059381 3975 mvcc.cc:205] Tried to move back new op lower bound from 7024682016993542144 to 7024682016893399040. Current Snapshot: MvccSnapshot[applied={T|T < 7024682016993542144}]
I20240506 15:44:18.060323 3976 mvcc.cc:205] Tried to move back new op lower bound from 7024682016993542144 to 7024682016893399040. Current Snapshot: MvccSnapshot[applied={T|T < 7024682016993542144 or (T in {7024682016993542144})}]
I20240506 15:44:18.060446 3977 mvcc.cc:205] Tried to move back new op lower bound from 7024682016993542144 to 7024682016893399040. Current Snapshot: MvccSnapshot[applied={T|T < 7024682016993542144 or (T in {7024682016993542144})}]
I20240506 15:44:18.061441 3494 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36630:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:18.061534 3494 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36630:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:18.062304 3494 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e53d4c715f374bbab9e7a3b7ebf0e2e5: Sending DeleteTablet for 3 replicas of tablet a5ed8721c03244b199918a1a24bdc5bd
I20240506 15:44:18.062620 3882 tablet_service.cc:1514] Processing DeleteTablet for tablet a5ed8721c03244b199918a1a24bdc5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:18 UTC) from {username='slave'} at 127.0.0.1:40516
I20240506 15:44:18.062644 3620 tablet_service.cc:1514] Processing DeleteTablet for tablet a5ed8721c03244b199918a1a24bdc5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:18 UTC) from {username='slave'} at 127.0.0.1:54976
I20240506 15:44:18.062927 3751 tablet_service.cc:1514] Processing DeleteTablet for tablet a5ed8721c03244b199918a1a24bdc5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:18 UTC) from {username='slave'} at 127.0.0.1:55794
I20240506 15:44:18.063051 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 3558
I20240506 15:44:18.063118 3750 tablet_service.cc:1514] Processing DeleteTablet for tablet a5ed8721c03244b199918a1a24bdc5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:18 UTC) from {username='slave'} at 127.0.0.1:55794
I20240506 15:44:18.063354 3986 tablet_replica.cc:330] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160: stopping tablet replica
I20240506 15:44:18.063444 3986 raft_consensus.cc:2229] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:44:18.063558 3986 raft_consensus.cc:2258] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:18.064368 3984 tablet_replica.cc:330] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: stopping tablet replica
I20240506 15:44:18.064458 3984 raft_consensus.cc:2229] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:44:18.064548 3984 raft_consensus.cc:2258] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:18.064939 3984 ts_tablet_manager.cc:1875] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:44:18.066439 3984 ts_tablet_manager.cc:1888] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:44:18.066520 3984 log.cc:1196] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/a5ed8721c03244b199918a1a24bdc5bd
I20240506 15:44:18.066767 3984 ts_tablet_manager.cc:1909] T a5ed8721c03244b199918a1a24bdc5bd P ca04530d62ee4d1a9879f03e06884069: Deleting consensus metadata
I20240506 15:44:18.067137 3482 catalog_manager.cc:4830] TS ca04530d62ee4d1a9879f03e06884069 (127.17.240.195:36093): tablet a5ed8721c03244b199918a1a24bdc5bd (table test-table [id=39159b9c2ebc46f1a35f15468ae5c86f]) successfully deleted
W20240506 15:44:18.070394 3485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:46647 (error 108)
I20240506 15:44:18.070852 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:18.070737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:46647 (local address 127.0.0.1:54984)
0506 15:44:18.070793 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:18.070824 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20240506 15:44:18.070895 3986 ts_tablet_manager.cc:1875] T a5ed8721c03244b199918a1a24bdc5bd P 6b8712cab0ef43ceb8a639e717a39160: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 15:44:18.070998 3485 catalog_manager.cc:4573] TS 8c3ca88c73464b978dacfa6691bc6eec (127.17.240.193:46647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5ed8721c03244b199918a1a24bdc5bd: Network error: Client connection negotiation failed: client connection to 127.17.240.193:46647: connect: Connection refused (error 111)
I20240506 15:44:18.071204 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 3689
I20240506 15:44:18.077348 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 3820
I20240506 15:44:18.077366 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:18.077246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:35121 (local address 127.0.0.1:55824)
0506 15:44:18.077296 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:18.077329 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.194:35121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20240506 15:44:18.081758 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 3467
2024-05-06T15:44:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (772 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:18 dist-test-slave-v0k1 krb5kdc[3990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:18 dist-test-slave-v0k1 krb5kdc[3990](info): set up 2 sockets
May 06 15:44:18 dist-test-slave-v0k1 krb5kdc[3990](info): commencing operation
krb5kdc: starting...
W20240506 15:44:19.042846 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:19 dist-test-slave-v0k1 krb5kdc[3990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:19.159389 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:46871
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:46871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:19.241989 4003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:19.242173 4003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:19.242206 4003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:19.243467 4003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:19.243508 4003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:19.243526 4003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:19.243546 4003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:19.243564 4003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:19.245065 4003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:46871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:46871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:19.245280 4003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:19.245460 4003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:19.249974 4008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:19.250640 4009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:19.251252 4011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:19.251832 4003 server_base.cc:1034] running on GCE node
I20240506 15:44:19.252085 4003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:19.252322 4003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:19.253468 4003 hybrid_clock.cc:648] HybridClock initialized: now 1715010259253449 us; error 35 us; skew 500 ppm
May 06 15:44:19 dist-test-slave-v0k1 krb5kdc[3990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010259, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:19.255790 4003 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:19.256193 4003 webserver.cc:460] Webserver started at http://127.17.240.254:38939/ using document root <none> and password file <none>
I20240506 15:44:19.256381 4003 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:19.256433 4003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:19.256546 4003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:19.257522 4003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "3479b3f12b944b7281859c65ea1793df"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "2d5f1d7667cc686f651df0a898bad1fd"
server_key_iv: "ccb06fb84826c4d9a139e0083f6564b7"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.257824 4003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "3479b3f12b944b7281859c65ea1793df"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "2d5f1d7667cc686f651df0a898bad1fd"
server_key_iv: "ccb06fb84826c4d9a139e0083f6564b7"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.258930 4003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:44:19.259577 4018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.259721 4003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.259792 4003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "3479b3f12b944b7281859c65ea1793df"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "2d5f1d7667cc686f651df0a898bad1fd"
server_key_iv: "ccb06fb84826c4d9a139e0083f6564b7"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.259851 4003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:19.283242 4003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:19.285068 4003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:19.285192 4003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:19.289083 4003 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:46871
I20240506 15:44:19.289129 4070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:46871 every 8 connection(s)
I20240506 15:44:19.289441 4003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:19.290375 4071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:19.292521 4071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df: Bootstrap starting.
I20240506 15:44:19.293056 4071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:19.293311 4071 log.cc:824] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:19.293937 4071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df: No bootstrap required, opened a new log
I20240506 15:44:19.293949 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 4003
I20240506 15:44:19.294052 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:19.294207 18371 external_mini_cluster.cc:1428] Setting key 0775375c4de642454f37da82b290fbd7
I20240506 15:44:19.295673 4071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3479b3f12b944b7281859c65ea1793df" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46871 } }
I20240506 15:44:19.295789 4071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:19.295817 4071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3479b3f12b944b7281859c65ea1793df, State: Initialized, Role: FOLLOWER
I20240506 15:44:19.295944 4071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3479b3f12b944b7281859c65ea1793df" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46871 } }
I20240506 15:44:19.296018 4071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:19.296048 4071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:19.296100 4071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:19.296609 4071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3479b3f12b944b7281859c65ea1793df" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46871 } }
I20240506 15:44:19.296702 4071 leader_election.cc:302] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3479b3f12b944b7281859c65ea1793df; no voters:
I20240506 15:44:19.296862 4071 leader_election.cc:288] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:19.296927 4076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:19.297053 4076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [term 1 LEADER]: Becoming Leader. State: Replica: 3479b3f12b944b7281859c65ea1793df, State: Running, Role: LEADER
I20240506 15:44:19.297127 4071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:19.297204 4076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3479b3f12b944b7281859c65ea1793df" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46871 } }
I20240506 15:44:19.297557 4076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3479b3f12b944b7281859c65ea1793df. Latest consensus state: current_term: 1 leader_uuid: "3479b3f12b944b7281859c65ea1793df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3479b3f12b944b7281859c65ea1793df" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46871 } } }
I20240506 15:44:19.297570 4077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3479b3f12b944b7281859c65ea1793df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3479b3f12b944b7281859c65ea1793df" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46871 } } }
I20240506 15:44:19.297706 4076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:19.297925 4077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:19.298671 4085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:19.299180 4085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:19.300678 4085 catalog_manager.cc:1347] Generated new cluster ID: 479f0b4fe8ca4ff89a8b2b9f7f501004
I20240506 15:44:19.300732 4085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:44:19 dist-test-slave-v0k1 krb5kdc[3990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:19.305510 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.294706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.17.240.254:46871)
0506 15:44:19.294866 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.294875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.294893 (+ 18us) server_negotiation.cc:407] Connection header received
0506 15:44:19.295652 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.295660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.295708 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.295853 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:19.296816 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.298436 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.299323 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.299544 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.302092 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:19.302114 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:19.302115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:19.302132 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:19.304164 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:19.304552 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:19.304554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:19.304555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:19.304603 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:19.304749 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:19.304750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:19.304750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:19.304959 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:19.305077 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.305206 (+ 129us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.305320 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20240506 15:44:19.320335 4085 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:19.321000 4085 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:19.329310 4085 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df: Generated new TSK 0
I20240506 15:44:19.329507 4085 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:44:19.359822 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46871
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:19.440702 4097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:19.440898 4097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:19.440923 4097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:19.442250 4097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:19.442292 4097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:19.442320 4097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:44:19.443912 4097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:19.444108 4097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:19.444303 4097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:19.449461 4105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:19.449898 4102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:19.450676 4103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:19.451359 4097 server_base.cc:1034] running on GCE node
I20240506 15:44:19.451556 4097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:19.451786 4097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:19.452924 4097 hybrid_clock.cc:648] HybridClock initialized: now 1715010259452907 us; error 41 us; skew 500 ppm
May 06 15:44:19 dist-test-slave-v0k1 krb5kdc[3990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010259, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:19.455508 4097 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:44:19.455931 4097 webserver.cc:460] Webserver started at http://127.17.240.193:35909/ using document root <none> and password file <none>
I20240506 15:44:19.456116 4097 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:19.456161 4097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:19.456269 4097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:19.457300 4097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "16467663109e4ab9b84591fd83ff1f43"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "4b7413dcda5e96229caf5429df964fb5"
server_key_iv: "3f11006fa4978cf898e1ee24ee0e7301"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.457623 4097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "16467663109e4ab9b84591fd83ff1f43"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "4b7413dcda5e96229caf5429df964fb5"
server_key_iv: "3f11006fa4978cf898e1ee24ee0e7301"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.458844 4097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:44:19.459609 4112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.459772 4097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.459852 4097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "16467663109e4ab9b84591fd83ff1f43"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "4b7413dcda5e96229caf5429df964fb5"
server_key_iv: "3f11006fa4978cf898e1ee24ee0e7301"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.459919 4097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:19.480587 4097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:19.482602 4097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:19.482728 4097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:19.482941 4097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:19.483304 4097 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:19.483340 4097 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.483376 4097 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:19.483407 4097 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.489033 4097 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:34175
I20240506 15:44:19.489085 4225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:34175 every 8 connection(s)
I20240506 15:44:19.489378 4097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
May 06 15:44:19 dist-test-slave-v0k1 krb5kdc[3990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010259, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:19.494696 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 4097
I20240506 15:44:19.494824 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:44:19.495049 18371 external_mini_cluster.cc:1428] Setting key 615e39f6f074bc08b6857e03f5bc659f
I20240506 15:44:19.498265 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.490102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:34421 (local address 127.17.240.254:46871)
0506 15:44:19.490186 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.490189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.490379 (+ 190us) server_negotiation.cc:407] Connection header received
0506 15:44:19.490811 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.490812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.490826 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.490891 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:19.491988 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.492489 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.493021 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.493127 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.495316 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:19.495322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:19.495322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:19.495330 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:19.496836 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:19.497327 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:19.497329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:19.497329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:19.497358 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:19.497554 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:19.497554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:19.497555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:19.497684 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:19.497754 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.498012 (+ 258us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.498116 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240506 15:44:19.498276 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.490052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:46871 (local address 127.17.240.193:34421)
0506 15:44:19.490254 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:19.490281 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:19.490724 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:19.490944 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:19.490947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:19.491275 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:19.491894 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.491899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.492569 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.492570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:19.492955 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.492956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.493070 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.493624 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:19.493639 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:19.495203 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:19.496915 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:19.496923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:19.496926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:19.497243 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:19.497423 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:19.497423 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:19.497424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:19.497500 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:19.497750 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:19.497755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:19.497869 (+ 114us) client_negotiation.cc:769] Sending connection context
0506 15:44:19.497963 (+ 94us) client_negotiation.cc:240] Negotiation successful
0506 15:44:19.498129 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":63,"threads_started":1}
I20240506 15:44:19.498832 4226 heartbeater.cc:347] Connected to a master server at 127.17.240.254:46871
I20240506 15:44:19.498937 4226 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:19.499173 4226 heartbeater.cc:511] Master 127.17.240.254:46871 requested a full tablet report, sending...
I20240506 15:44:19.499727 4035 ts_manager.cc:194] Registered new tserver with Master: 16467663109e4ab9b84591fd83ff1f43 (127.17.240.193:34175)
I20240506 15:44:19.500345 4035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:34421
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:44:19.522501 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46871
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:19.603114 4231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:19.603318 4231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:19.603364 4231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:19.604712 4231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:19.604760 4231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:19.604800 4231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:44:19.606405 4231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:19.606657 4231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:19.606849 4231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:19.611884 4239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:19.612507 4237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:19.613198 4236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:19.613869 4231 server_base.cc:1034] running on GCE node
I20240506 15:44:19.614048 4231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:19.614251 4231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:19.615378 4231 hybrid_clock.cc:648] HybridClock initialized: now 1715010259615380 us; error 77 us; skew 500 ppm
May 06 15:44:19 dist-test-slave-v0k1 krb5kdc[3990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010259, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:19.617985 4231 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:44:19.618403 4231 webserver.cc:460] Webserver started at http://127.17.240.194:38435/ using document root <none> and password file <none>
I20240506 15:44:19.618587 4231 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:19.618628 4231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:19.618734 4231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:19.619788 4231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "68e3ebc7d10c42d0aa62de7c92422676"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "c660f77e96ce7830ade0a004f2a24f9d"
server_key_iv: "21c7a105c77350910dfb581744f6cf71"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.620105 4231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "68e3ebc7d10c42d0aa62de7c92422676"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "c660f77e96ce7830ade0a004f2a24f9d"
server_key_iv: "21c7a105c77350910dfb581744f6cf71"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.621345 4231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:44:19.622073 4246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.622227 4231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:19.622311 4231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "68e3ebc7d10c42d0aa62de7c92422676"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "c660f77e96ce7830ade0a004f2a24f9d"
server_key_iv: "21c7a105c77350910dfb581744f6cf71"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.622409 4231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:19.634702 4231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:19.636683 4231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:19.636822 4231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:19.637056 4231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:19.637380 4231 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:19.637418 4231 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.637452 4231 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:19.637477 4231 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.643282 4231 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:39067
I20240506 15:44:19.643412 4359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:39067 every 8 connection(s)
I20240506 15:44:19.643651 4231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:44:19.646941 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 4231
I20240506 15:44:19.647073 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:44:19.647257 18371 external_mini_cluster.cc:1428] Setting key ec4add54bce4521a87ca8a2ed88865b7
May 06 15:44:19 dist-test-slave-v0k1 krb5kdc[3990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010259, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:19.652449 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.644396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:44469 (local address 127.17.240.254:46871)
0506 15:44:19.644465 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.644467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.644623 (+ 156us) server_negotiation.cc:407] Connection header received
0506 15:44:19.645075 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.645077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.645086 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.645148 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:19.646271 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.646785 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.647524 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.647691 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.649686 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:19.649690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:19.649690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:19.649697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:19.651249 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:19.651643 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:19.651644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:19.651645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:19.651670 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:19.651842 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:19.651843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:19.651843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:19.651945 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:19.652008 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.652249 (+ 241us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.652355 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240506 15:44:19.652554 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.644358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:46871 (local address 127.17.240.194:44469)
0506 15:44:19.644521 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:19.644545 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:19.644987 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:19.645203 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:19.645206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:19.645540 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:19.646158 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.646165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.646900 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.646901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:19.647397 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.647401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.647562 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.648122 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:19.648137 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:19.649588 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:19.651338 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:19.651345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:19.651348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:19.651568 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:19.651730 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:19.651730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:19.651731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:19.651788 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:19.652003 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:19.652008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:19.652125 (+ 117us) client_negotiation.cc:769] Sending connection context
0506 15:44:19.652225 (+ 100us) client_negotiation.cc:240] Negotiation successful
0506 15:44:19.652365 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20240506 15:44:19.653002 4360 heartbeater.cc:347] Connected to a master server at 127.17.240.254:46871
I20240506 15:44:19.653121 4360 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:19.653342 4360 heartbeater.cc:511] Master 127.17.240.254:46871 requested a full tablet report, sending...
I20240506 15:44:19.653735 4035 ts_manager.cc:194] Registered new tserver with Master: 68e3ebc7d10c42d0aa62de7c92422676 (127.17.240.194:39067)
I20240506 15:44:19.654111 4035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:44469
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:44:19.672061 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46871
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:19.753424 4365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:19.753620 4365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:19.753643 4365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:19.754906 4365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:19.754947 4365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:19.754971 4365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:44:19.756593 4365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:46871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:19.756795 4365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:19.756961 4365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:19.761745 4373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:19.762305 4371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:19.762965 4370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:19.763695 4365 server_base.cc:1034] running on GCE node
I20240506 15:44:19.763865 4365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:19.764060 4365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:19.765201 4365 hybrid_clock.cc:648] HybridClock initialized: now 1715010259765184 us; error 35 us; skew 500 ppm
May 06 15:44:19 dist-test-slave-v0k1 krb5kdc[3990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010259, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:19.767577 4365 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:44:19.767983 4365 webserver.cc:460] Webserver started at http://127.17.240.195:38611/ using document root <none> and password file <none>
I20240506 15:44:19.768146 4365 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:19.768182 4365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:19.768266 4365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:19.769181 4365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "b5696bd3f19d4236b8d120036d3798cd"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "c8f3dd5a26574e8b3a0fbd3a0e6abc8c"
server_key_iv: "53446f5712718d2ac3edc026ef14c5e1"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.769443 4365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "b5696bd3f19d4236b8d120036d3798cd"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "c8f3dd5a26574e8b3a0fbd3a0e6abc8c"
server_key_iv: "53446f5712718d2ac3edc026ef14c5e1"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.770576 4365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:44:19.771207 4380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.771373 4365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.771449 4365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "b5696bd3f19d4236b8d120036d3798cd"
format_stamp: "Formatted at 2024-05-06 15:44:19 on dist-test-slave-v0k1"
server_key: "c8f3dd5a26574e8b3a0fbd3a0e6abc8c"
server_key_iv: "53446f5712718d2ac3edc026ef14c5e1"
server_key_version: "encryptionkey@0"
I20240506 15:44:19.771520 4365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:19.790858 4365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:19.792584 4365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:19.792706 4365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:19.792913 4365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:19.793263 4365 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:19.793298 4365 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.793331 4365 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:19.793350 4365 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:19.798867 4365 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:44863
I20240506 15:44:19.798921 4493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:44863 every 8 connection(s)
I20240506 15:44:19.799266 4365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
May 06 15:44:19 dist-test-slave-v0k1 krb5kdc[3990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010259, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:19.806725 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 4365
I20240506 15:44:19.806828 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:44:19.807001 18371 external_mini_cluster.cc:1428] Setting key e2d9f7700c7d64a110259710244096a6
I20240506 15:44:19.807745 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.799947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:39255 (local address 127.17.240.254:46871)
0506 15:44:19.799990 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.799992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.800177 (+ 185us) server_negotiation.cc:407] Connection header received
0506 15:44:19.800665 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.800667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.800679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.800738 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:19.801775 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.802277 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.802854 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.803062 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.804929 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:19.804934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:19.804935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:19.804941 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:19.806329 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:19.806690 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:19.806691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:19.806691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:19.806718 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:19.806943 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:19.806944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:19.806944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:19.807099 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:19.807171 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.807563 (+ 392us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.807658 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:44:19.807781 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.799908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:46871 (local address 127.17.240.195:39255)
0506 15:44:19.800071 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:19.800097 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:19.800563 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:19.800808 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:19.800811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:19.801112 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:19.801679 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.801685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.802365 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.802367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:19.802777 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.802779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.802911 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.803479 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:19.803494 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:19.804837 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:19.806406 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:19.806410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:19.806413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:19.806616 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:19.806794 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:19.806795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:19.806795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:19.806875 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:19.807167 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:19.807173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:19.807327 (+ 154us) client_negotiation.cc:769] Sending connection context
0506 15:44:19.807450 (+ 123us) client_negotiation.cc:240] Negotiation successful
0506 15:44:19.807600 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20240506 15:44:19.808216 4494 heartbeater.cc:347] Connected to a master server at 127.17.240.254:46871
I20240506 15:44:19.808310 4494 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:19.808506 4494 heartbeater.cc:511] Master 127.17.240.254:46871 requested a full tablet report, sending...
I20240506 15:44:19.808876 4035 ts_manager.cc:194] Registered new tserver with Master: b5696bd3f19d4236b8d120036d3798cd (127.17.240.195:44863)
I20240506 15:44:19.809230 4035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:39255
I20240506 15:44:19.818991 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:44:19.825911 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.820360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.17.240.254:46871)
0506 15:44:19.820458 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.820460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.820536 (+ 76us) server_negotiation.cc:407] Connection header received
0506 15:44:19.820580 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.820580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.820588 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.820639 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:19.821268 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.821744 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.822257 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.822355 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.823513 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:19.823516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:19.823516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:19.823522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:19.824913 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:19.825298 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:19.825299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:19.825299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:19.825322 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:19.825475 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:19.825476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:19.825476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:19.825579 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:19.825660 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.825752 (+ 92us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.825799 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240506 15:44:19.827702 4035 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:44:19.829185 4035 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:44:19.834251 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.831436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:44863 (local address 127.0.0.1:40344)
0506 15:44:19.831550 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:19.831575 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:19.831646 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:19.831991 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:19.831994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:19.831996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:19.832269 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.832272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.833337 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.833338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:19.833956 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.833957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.834035 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.834075 (+ 40us) client_negotiation.cc:769] Sending connection context
0506 15:44:19.834126 (+ 51us) client_negotiation.cc:240] Negotiation successful
0506 15:44:19.834170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":45,"threads_started":1}
I20240506 15:44:19.834957 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.831634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:34175 (local address 127.0.0.1:52896)
0506 15:44:19.832140 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:19.832148 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:19.832201 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:19.832453 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:19.832454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:19.832456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:19.832653 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.832655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.833939 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.833940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:19.834751 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.834754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.834818 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.834822 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:44:19.834854 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:44:19.834893 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":56,"threads_started":1}
I20240506 15:44:19.835064 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.831531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.17.240.195:44863)
0506 15:44:19.831785 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.831794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.831805 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:44:19.831849 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.831850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.831868 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.831978 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:19.832375 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.833253 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.834069 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.834833 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.834883 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.834917 (+ 34us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.834959 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
I20240506 15:44:19.835675 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.831742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52896 (local address 127.17.240.193:34175)
0506 15:44:19.832094 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.832101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.832223 (+ 122us) server_negotiation.cc:407] Connection header received
0506 15:44:19.832269 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.832271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.832290 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.832369 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:19.832722 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.833851 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.834850 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.835477 (+ 627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.835520 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.835554 (+ 34us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.835595 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":90,"threads_started":1}
I20240506 15:44:19.835700 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.831776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:39067 (local address 127.0.0.1:44512)
0506 15:44:19.832254 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:19.832261 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:19.832309 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:19.832655 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:19.832656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:19.832658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:19.832878 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.832881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.834160 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.834161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:19.835491 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.835494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.835562 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.835566 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:44:19.835599 (+ 33us) client_negotiation.cc:240] Negotiation successful
0506 15:44:19.835635 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":74,"threads_started":1}
I20240506 15:44:19.835697 4428 tablet_service.cc:1467] Processing CreateTablet for tablet fcaa78110ee143b99c32614336fa3af1 (DEFAULT_TABLE table=test-table [id=0977e535be0141e1983f4e047d06b000]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:19.836011 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcaa78110ee143b99c32614336fa3af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:19.836323 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.831908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.17.240.194:39067)
0506 15:44:19.832090 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.832098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.832432 (+ 334us) server_negotiation.cc:407] Connection header received
0506 15:44:19.832477 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.832479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.832498 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.832573 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:19.832982 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.834075 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.835599 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.836126 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.836167 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.836199 (+ 32us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.836247 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20240506 15:44:19.836251 4160 tablet_service.cc:1467] Processing CreateTablet for tablet fcaa78110ee143b99c32614336fa3af1 (DEFAULT_TABLE table=test-table [id=0977e535be0141e1983f4e047d06b000]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:19.836511 4160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcaa78110ee143b99c32614336fa3af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:19.836894 4294 tablet_service.cc:1467] Processing CreateTablet for tablet fcaa78110ee143b99c32614336fa3af1 (DEFAULT_TABLE table=test-table [id=0977e535be0141e1983f4e047d06b000]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:19.837144 4294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcaa78110ee143b99c32614336fa3af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:19.838563 4511 tablet_bootstrap.cc:492] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: Bootstrap starting.
I20240506 15:44:19.839170 4511 tablet_bootstrap.cc:654] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:19.839453 4511 log.cc:824] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:19.839687 4512 tablet_bootstrap.cc:492] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43: Bootstrap starting.
I20240506 15:44:19.840195 4514 tablet_bootstrap.cc:492] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: Bootstrap starting.
I20240506 15:44:19.840363 4512 tablet_bootstrap.cc:654] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:19.840703 4514 tablet_bootstrap.cc:654] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:19.840687 4512 log.cc:824] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:19.840752 4511 tablet_bootstrap.cc:492] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: No bootstrap required, opened a new log
I20240506 15:44:19.840803 4511 ts_tablet_manager.cc:1367] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:19.841010 4514 log.cc:824] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:19.841599 4514 tablet_bootstrap.cc:492] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: No bootstrap required, opened a new log
I20240506 15:44:19.841657 4514 ts_tablet_manager.cc:1367] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:19.842398 4511 raft_consensus.cc:348] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 } } peers { permanent_uuid: "16467663109e4ab9b84591fd83ff1f43" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34175 } } peers { permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 } }
I20240506 15:44:19.842518 4511 raft_consensus.cc:374] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:19.842535 4511 raft_consensus.cc:729] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5696bd3f19d4236b8d120036d3798cd, State: Initialized, Role: FOLLOWER
I20240506 15:44:19.842622 4511 consensus_queue.cc:260] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 } } peers { permanent_uuid: "16467663109e4ab9b84591fd83ff1f43" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34175 } } peers { permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 } }
I20240506 15:44:19.842856 4511 ts_tablet_manager.cc:1398] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:44:19.842922 4494 heartbeater.cc:503] Master 127.17.240.254:46871 was elected leader, sending a full tablet report...
I20240506 15:44:19.843508 4512 tablet_bootstrap.cc:492] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43: No bootstrap required, opened a new log
I20240506 15:44:19.843585 4512 ts_tablet_manager.cc:1367] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240506 15:44:19.844362 4514 raft_consensus.cc:348] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 } } peers { permanent_uuid: "16467663109e4ab9b84591fd83ff1f43" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34175 } } peers { permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 } }
I20240506 15:44:19.844513 4514 raft_consensus.cc:374] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:19.844530 4514 raft_consensus.cc:729] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68e3ebc7d10c42d0aa62de7c92422676, State: Initialized, Role: FOLLOWER
I20240506 15:44:19.844625 4514 consensus_queue.cc:260] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 } } peers { permanent_uuid: "16467663109e4ab9b84591fd83ff1f43" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34175 } } peers { permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 } }
I20240506 15:44:19.844887 4514 ts_tablet_manager.cc:1398] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:19.844908 4360 heartbeater.cc:503] Master 127.17.240.254:46871 was elected leader, sending a full tablet report...
I20240506 15:44:19.845669 4512 raft_consensus.cc:348] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 } } peers { permanent_uuid: "16467663109e4ab9b84591fd83ff1f43" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34175 } } peers { permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 } }
I20240506 15:44:19.845814 4512 raft_consensus.cc:374] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:19.845844 4512 raft_consensus.cc:729] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16467663109e4ab9b84591fd83ff1f43, State: Initialized, Role: FOLLOWER
I20240506 15:44:19.845948 4512 consensus_queue.cc:260] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 } } peers { permanent_uuid: "16467663109e4ab9b84591fd83ff1f43" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34175 } } peers { permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 } }
I20240506 15:44:19.846168 4512 ts_tablet_manager.cc:1398] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:19.846235 4226 heartbeater.cc:503] Master 127.17.240.254:46871 was elected leader, sending a full tablet report...
W20240506 15:44:19.894253 4361 tablet.cc:2369] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:44:19.896479 4519 raft_consensus.cc:482] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:19.896587 4519 raft_consensus.cc:504] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 } } peers { permanent_uuid: "16467663109e4ab9b84591fd83ff1f43" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34175 } } peers { permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 } }
I20240506 15:44:19.896909 4519 leader_election.cc:288] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5696bd3f19d4236b8d120036d3798cd (127.17.240.195:44863), 68e3ebc7d10c42d0aa62de7c92422676 (127.17.240.194:39067)
I20240506 15:44:19.899919 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.897118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:44863 (local address 127.17.240.193:53813)
0506 15:44:19.897198 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:19.897212 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:19.897277 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:19.897424 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:19.897425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:19.897427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:19.897696 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.897699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.898664 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.898665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:19.899652 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.899655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.899736 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.899739 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:44:19.899774 (+ 35us) client_negotiation.cc:240] Negotiation successful
0506 15:44:19.899811 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20240506 15:44:19.899928 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.897230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:39067 (local address 127.17.240.193:51507)
0506 15:44:19.897432 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:19.897441 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:19.897480 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:19.897621 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:19.897622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:19.897623 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:19.897883 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.897884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.898721 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.898721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:19.899653 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:19.899655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.899734 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.899739 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:44:19.899774 (+ 35us) client_negotiation.cc:240] Negotiation successful
0506 15:44:19.899811 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":130,"threads_started":1}
I20240506 15:44:19.900316 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.897473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:51507 (local address 127.17.240.194:39067)
0506 15:44:19.897515 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.897517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.897525 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:44:19.897548 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.897549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.897556 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.897608 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:19.897949 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.898636 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.899734 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.900191 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.900200 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.900222 (+ 22us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.900251 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:44:19.900317 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.897220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:53813 (local address 127.17.240.195:44863)
0506 15:44:19.897274 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.897276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.897288 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:44:19.897339 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.897340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.897350 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.897407 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:19.897778 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.898591 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.899741 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.900198 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.900204 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.900224 (+ 20us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.900252 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:44:19.900660 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcaa78110ee143b99c32614336fa3af1" candidate_uuid: "16467663109e4ab9b84591fd83ff1f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5696bd3f19d4236b8d120036d3798cd" is_pre_election: true
I20240506 15:44:19.900702 4314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcaa78110ee143b99c32614336fa3af1" candidate_uuid: "16467663109e4ab9b84591fd83ff1f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68e3ebc7d10c42d0aa62de7c92422676" is_pre_election: true
I20240506 15:44:19.900825 4448 raft_consensus.cc:2454] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16467663109e4ab9b84591fd83ff1f43 in term 0.
I20240506 15:44:19.900825 4314 raft_consensus.cc:2454] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16467663109e4ab9b84591fd83ff1f43 in term 0.
I20240506 15:44:19.901049 4113 leader_election.cc:302] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16467663109e4ab9b84591fd83ff1f43, 68e3ebc7d10c42d0aa62de7c92422676; no voters:
I20240506 15:44:19.901182 4519 raft_consensus.cc:2781] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:44:19.901265 4519 raft_consensus.cc:482] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:44:19.901298 4519 raft_consensus.cc:3037] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:19.901962 4519 raft_consensus.cc:504] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 } } peers { permanent_uuid: "16467663109e4ab9b84591fd83ff1f43" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34175 } } peers { permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 } }
I20240506 15:44:19.902096 4519 leader_election.cc:288] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [CANDIDATE]: Term 1 election: Requested vote from peers b5696bd3f19d4236b8d120036d3798cd (127.17.240.195:44863), 68e3ebc7d10c42d0aa62de7c92422676 (127.17.240.194:39067)
I20240506 15:44:19.902267 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcaa78110ee143b99c32614336fa3af1" candidate_uuid: "16467663109e4ab9b84591fd83ff1f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5696bd3f19d4236b8d120036d3798cd"
I20240506 15:44:19.902300 4314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcaa78110ee143b99c32614336fa3af1" candidate_uuid: "16467663109e4ab9b84591fd83ff1f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68e3ebc7d10c42d0aa62de7c92422676"
I20240506 15:44:19.902343 4448 raft_consensus.cc:3037] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:19.902377 4314 raft_consensus.cc:3037] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:19.903052 4448 raft_consensus.cc:2454] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16467663109e4ab9b84591fd83ff1f43 in term 1.
I20240506 15:44:19.903105 4314 raft_consensus.cc:2454] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16467663109e4ab9b84591fd83ff1f43 in term 1.
I20240506 15:44:19.903221 4113 leader_election.cc:302] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16467663109e4ab9b84591fd83ff1f43, b5696bd3f19d4236b8d120036d3798cd; no voters:
I20240506 15:44:19.903324 4519 raft_consensus.cc:2781] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:19.903471 4519 raft_consensus.cc:686] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [term 1 LEADER]: Becoming Leader. State: Replica: 16467663109e4ab9b84591fd83ff1f43, State: Running, Role: LEADER
I20240506 15:44:19.903558 4519 consensus_queue.cc:237] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 } } peers { permanent_uuid: "16467663109e4ab9b84591fd83ff1f43" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34175 } } peers { permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 } }
I20240506 15:44:19.904297 4035 catalog_manager.cc:5484] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16467663109e4ab9b84591fd83ff1f43 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "16467663109e4ab9b84591fd83ff1f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16467663109e4ab9b84591fd83ff1f43" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 } health_report { overall_health: UNKNOWN } } }
I20240506 15:44:19.909300 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.907221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.17.240.193:34175)
0506 15:44:19.907278 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:19.907280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:19.907288 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:44:19.907323 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:19.907324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:19.907338 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:19.907402 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:19.907627 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.908108 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:19.908670 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:19.908824 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:19.908971 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:19.909055 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:19.909136 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:19.909222 (+ 86us) server_negotiation.cc:299] Negotiation successful
0506 15:44:19.909230 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240506 15:44:19.910867 4448 raft_consensus.cc:1261] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd [term 1 FOLLOWER]: Refusing update from remote peer 16467663109e4ab9b84591fd83ff1f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:19.910885 4314 raft_consensus.cc:1261] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676 [term 1 FOLLOWER]: Refusing update from remote peer 16467663109e4ab9b84591fd83ff1f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:19.911206 4521 consensus_queue.cc:1035] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68e3ebc7d10c42d0aa62de7c92422676" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 39067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:19.911338 4519 consensus_queue.cc:1035] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5696bd3f19d4236b8d120036d3798cd" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 44863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:19.916232 4035 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34138:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:19.916326 4035 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34138:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:19.916980 4035 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 3479b3f12b944b7281859c65ea1793df: Sending DeleteTablet for 3 replicas of tablet fcaa78110ee143b99c32614336fa3af1
I20240506 15:44:19.917240 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet fcaa78110ee143b99c32614336fa3af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:19 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:40344
I20240506 15:44:19.917300 4160 tablet_service.cc:1514] Processing DeleteTablet for tablet fcaa78110ee143b99c32614336fa3af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:19 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:52896
I20240506 15:44:19.917348 4294 tablet_service.cc:1514] Processing DeleteTablet for tablet fcaa78110ee143b99c32614336fa3af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:19 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:44512
I20240506 15:44:19.917523 4535 tablet_replica.cc:330] T fcaa78110ee143b99c32614336fa3af1 P 16467663109e4ab9b84591fd83ff1f43: stopping tablet replica
I20240506 15:44:19.917551 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 4097
I20240506 15:44:19.918386 4536 tablet_replica.cc:330] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: stopping tablet replica
I20240506 15:44:19.918474 4536 raft_consensus.cc:2229] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:44:19.918565 4536 raft_consensus.cc:2258] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:19.918924 4536 ts_tablet_manager.cc:1875] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:44:19.919765 4534 tablet_replica.cc:330] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: stopping tablet replica
I20240506 15:44:19.919859 4534 raft_consensus.cc:2229] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:44:19.919955 4534 raft_consensus.cc:2258] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:19.920413 4534 ts_tablet_manager.cc:1875] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:44:19.920564 4536 ts_tablet_manager.cc:1888] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:44:19.920639 4536 log.cc:1196] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/fcaa78110ee143b99c32614336fa3af1
I20240506 15:44:19.920897 4536 ts_tablet_manager.cc:1909] T fcaa78110ee143b99c32614336fa3af1 P 68e3ebc7d10c42d0aa62de7c92422676: Deleting consensus metadata
I20240506 15:44:19.921257 4019 catalog_manager.cc:4830] TS 68e3ebc7d10c42d0aa62de7c92422676 (127.17.240.194:39067): tablet fcaa78110ee143b99c32614336fa3af1 (table test-table [id=0977e535be0141e1983f4e047d06b000]) successfully deleted
I20240506 15:44:19.922358 4534 ts_tablet_manager.cc:1888] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:44:19.922431 4534 log.cc:1196] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/fcaa78110ee143b99c32614336fa3af1
I20240506 15:44:19.922720 4534 ts_tablet_manager.cc:1909] T fcaa78110ee143b99c32614336fa3af1 P b5696bd3f19d4236b8d120036d3798cd: Deleting consensus metadata
I20240506 15:44:19.923110 4019 catalog_manager.cc:4830] TS b5696bd3f19d4236b8d120036d3798cd (127.17.240.195:44863): tablet fcaa78110ee143b99c32614336fa3af1 (table test-table [id=0977e535be0141e1983f4e047d06b000]) successfully deleted
W20240506 15:44:19.923943 4019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:34175 (error 108)
I20240506 15:44:19.924480 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:19.924322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:34175 (local address 127.0.0.1:52916)
0506 15:44:19.924387 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:19.924439 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20240506 15:44:19.924609 4019 catalog_manager.cc:4573] TS 16467663109e4ab9b84591fd83ff1f43 (127.17.240.193:34175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fcaa78110ee143b99c32614336fa3af1: Network error: Client connection negotiation failed: client connection to 127.17.240.193:34175: connect: Connection refused (error 111)
I20240506 15:44:19.925194 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 4231
I20240506 15:44:19.930605 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 4365
I20240506 15:44:19.935307 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 4003
2024-05-06T15:44:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1853 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2024-05-06T15:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:19Z Disabled control of system clock
I20240506 15:44:19.961460 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:33545
--webserver_interface=127.17.240.254
--webserver_port=0
--builtin_ntp_servers=127.17.240.212:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:33545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240506 15:44:20.042512 4541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:20.042721 4541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:20.042743 4541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:20.044234 4541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:20.044294 4541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:20.044306 4541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:20.044319 4541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:20.044333 4541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:20.046033 4541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:33545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:33545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:20.046298 4541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:20.046568 4541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:20.051486 4546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:20.052250 4547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:20.052855 4549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:20.053647 4541 server_base.cc:1034] running on GCE node
I20240506 15:44:20.053869 4541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:20.054170 4541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:20.055305 4541 hybrid_clock.cc:648] HybridClock initialized: now 1715010260055283 us; error 35 us; skew 500 ppm
I20240506 15:44:20.056543 4541 webserver.cc:460] Webserver started at http://127.17.240.254:44135/ using document root <none> and password file <none>
I20240506 15:44:20.056743 4541 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:20.056787 4541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:20.056941 4541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:20.057991 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "da241d71c29d40a09d6db164d334b0bb"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "ff24e7ff02d10d303b3c10f46e17c47d"
server_key_iv: "db41853e50fe75bf5597d44afbf8f5a1"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.058312 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "da241d71c29d40a09d6db164d334b0bb"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "ff24e7ff02d10d303b3c10f46e17c47d"
server_key_iv: "db41853e50fe75bf5597d44afbf8f5a1"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.059588 4541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:44:20.060248 4555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:20.060408 4541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:20.060496 4541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "da241d71c29d40a09d6db164d334b0bb"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "ff24e7ff02d10d303b3c10f46e17c47d"
server_key_iv: "db41853e50fe75bf5597d44afbf8f5a1"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.060562 4541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:20.076001 4541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:20.076292 4541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:20.076406 4541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:20.080122 4541 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:33545
I20240506 15:44:20.080155 4607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:33545 every 8 connection(s)
I20240506 15:44:20.080461 4541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:20.081075 4608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:20.083680 4608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb: Bootstrap starting.
I20240506 15:44:20.084264 4608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:20.084532 4608 log.cc:824] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:20.085193 4608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb: No bootstrap required, opened a new log
I20240506 15:44:20.085558 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 4541
I20240506 15:44:20.085644 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:20.085813 18371 external_mini_cluster.cc:1428] Setting key d50ecdd528fb271a11163ade443dee57
I20240506 15:44:20.086977 4608 raft_consensus.cc:348] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da241d71c29d40a09d6db164d334b0bb" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33545 } }
I20240506 15:44:20.087132 4608 raft_consensus.cc:374] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:20.087162 4608 raft_consensus.cc:729] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da241d71c29d40a09d6db164d334b0bb, State: Initialized, Role: FOLLOWER
I20240506 15:44:20.087265 4608 consensus_queue.cc:260] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da241d71c29d40a09d6db164d334b0bb" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33545 } }
I20240506 15:44:20.087337 4608 raft_consensus.cc:388] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:20.087396 4608 raft_consensus.cc:482] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:20.087455 4608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:20.088037 4608 raft_consensus.cc:504] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da241d71c29d40a09d6db164d334b0bb" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33545 } }
I20240506 15:44:20.088125 4608 leader_election.cc:302] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da241d71c29d40a09d6db164d334b0bb; no voters:
I20240506 15:44:20.088579 4608 leader_election.cc:288] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:20.088639 4613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:20.088794 4613 raft_consensus.cc:686] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [term 1 LEADER]: Becoming Leader. State: Replica: da241d71c29d40a09d6db164d334b0bb, State: Running, Role: LEADER
I20240506 15:44:20.088846 4608 sys_catalog.cc:564] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:20.088919 4613 consensus_queue.cc:237] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da241d71c29d40a09d6db164d334b0bb" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33545 } }
I20240506 15:44:20.089324 4613 sys_catalog.cc:455] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader da241d71c29d40a09d6db164d334b0bb. Latest consensus state: current_term: 1 leader_uuid: "da241d71c29d40a09d6db164d334b0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da241d71c29d40a09d6db164d334b0bb" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33545 } } }
I20240506 15:44:20.089339 4614 sys_catalog.cc:455] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da241d71c29d40a09d6db164d334b0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da241d71c29d40a09d6db164d334b0bb" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33545 } } }
I20240506 15:44:20.089532 4613 sys_catalog.cc:458] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:20.089627 4614 sys_catalog.cc:458] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:20.089825 4622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:20.090404 4622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:20.091095 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.086358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.17.240.254:33545)
0506 15:44:20.086537 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.086543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.086560 (+ 17us) server_negotiation.cc:407] Connection header received
0506 15:44:20.086979 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.086987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.087102 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.087253 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:20.087746 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.088609 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.089676 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.089953 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.090267 (+ 314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:20.090275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:20.090275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:20.090466 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:20.090499 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:20.090509 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:20.090583 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.090767 (+ 184us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.090844 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20240506 15:44:20.094743 4622 catalog_manager.cc:1347] Generated new cluster ID: 1e5647f6126942eb9196afd2258c47d2
I20240506 15:44:20.094802 4622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:20.116035 4622 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:20.116641 4622 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:20.120844 4622 catalog_manager.cc:5857] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb: Generated new TSK 0
I20240506 15:44:20.121038 4622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:20.127737 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33545
--builtin_ntp_servers=127.17.240.212:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240506 15:44:20.209820 4632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:20.210000 4632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:20.210021 4632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:20.211445 4632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:20.211495 4632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:20.211521 4632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:44:20.212965 4632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:20.213168 4632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:20.213421 4632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:20.218669 4637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:20.219054 4638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:20.219626 4640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:20.220500 4632 server_base.cc:1034] running on GCE node
I20240506 15:44:20.220700 4632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:20.220968 4632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:20.222155 4632 hybrid_clock.cc:648] HybridClock initialized: now 1715010260222145 us; error 31 us; skew 500 ppm
I20240506 15:44:20.223482 4632 webserver.cc:460] Webserver started at http://127.17.240.193:42683/ using document root <none> and password file <none>
I20240506 15:44:20.223680 4632 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:20.223726 4632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:20.223839 4632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:20.224813 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "d9354a260d4f477aa5fe7427654c8b82"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "cea8dc0113e903e2c9b3e00147f16588"
server_key_iv: "fcfa1fbb788c2334d303088b0d8ce3be"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.225126 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "d9354a260d4f477aa5fe7427654c8b82"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "cea8dc0113e903e2c9b3e00147f16588"
server_key_iv: "fcfa1fbb788c2334d303088b0d8ce3be"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.226291 4632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:44:20.226946 4646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:20.227128 4632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:20.227207 4632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "d9354a260d4f477aa5fe7427654c8b82"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "cea8dc0113e903e2c9b3e00147f16588"
server_key_iv: "fcfa1fbb788c2334d303088b0d8ce3be"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.227273 4632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:20.240108 4632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:20.240363 4632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:20.240473 4632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:20.240674 4632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:20.240971 4632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:20.241003 4632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:20.241039 4632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:20.241072 4632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:20.246500 4632 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:42489
I20240506 15:44:20.246567 4759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:42489 every 8 connection(s)
I20240506 15:44:20.246899 4632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:44:20.251307 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.247620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:37891 (local address 127.17.240.254:33545)
0506 15:44:20.247682 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.247684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.247901 (+ 217us) server_negotiation.cc:407] Connection header received
0506 15:44:20.248446 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.248448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.248461 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.248526 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:20.248956 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.249448 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.249956 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.250072 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.251030 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:20.251035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:20.251035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:20.251040 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:20.251054 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:20.251057 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:20.251106 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.251208 (+ 102us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.251237 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240506 15:44:20.251403 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.247593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:33545 (local address 127.17.240.193:37891)
0506 15:44:20.247790 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:20.247819 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:20.248263 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:20.248582 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:20.248586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:20.248593 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:20.248884 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.248887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.249521 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.249522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:20.249891 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.249892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.250004 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.250908 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:44:20.250909 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:20.250934 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:44:20.250951 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:20.251116 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:20.251119 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:20.251121 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:44:20.251199 (+ 78us) client_negotiation.cc:240] Negotiation successful
0506 15:44:20.251255 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20240506 15:44:20.251837 4760 heartbeater.cc:347] Connected to a master server at 127.17.240.254:33545
I20240506 15:44:20.251919 4760 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:20.252103 4760 heartbeater.cc:511] Master 127.17.240.254:33545 requested a full tablet report, sending...
I20240506 15:44:20.252566 4572 ts_manager.cc:194] Registered new tserver with Master: d9354a260d4f477aa5fe7427654c8b82 (127.17.240.193:42489)
I20240506 15:44:20.252693 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 4632
I20240506 15:44:20.252785 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:44:20.252959 18371 external_mini_cluster.cc:1428] Setting key e482f62b39c329c8e399ca2b6ddb4fa2
I20240506 15:44:20.253396 4572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.193:37891
I20240506 15:44:20.254104 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33545
--builtin_ntp_servers=127.17.240.212:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240506 15:44:20.334403 4763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:20.334619 4763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:20.334657 4763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:20.336148 4763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:20.336205 4763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:20.336256 4763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:44:20.337857 4763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:20.338124 4763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:20.338388 4763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:20.343797 4768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:20.344285 4771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:20.344712 4769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:20.345533 4763 server_base.cc:1034] running on GCE node
I20240506 15:44:20.345803 4763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:20.346041 4763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:20.347195 4763 hybrid_clock.cc:648] HybridClock initialized: now 1715010260347176 us; error 45 us; skew 500 ppm
I20240506 15:44:20.348661 4763 webserver.cc:460] Webserver started at http://127.17.240.194:34277/ using document root <none> and password file <none>
I20240506 15:44:20.348862 4763 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:20.348909 4763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:20.349049 4763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:20.350093 4763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "b8da924aa99343a283b4cd9b6eb3c31c"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "fb08b0fb23969a68121b0f0bd2a708b6"
server_key_iv: "b1dc61d84f9708ba7706cc9f7b0669a4"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.350445 4763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "b8da924aa99343a283b4cd9b6eb3c31c"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "fb08b0fb23969a68121b0f0bd2a708b6"
server_key_iv: "b1dc61d84f9708ba7706cc9f7b0669a4"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.351739 4763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:44:20.352468 4777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:20.352654 4763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:20.352737 4763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "b8da924aa99343a283b4cd9b6eb3c31c"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "fb08b0fb23969a68121b0f0bd2a708b6"
server_key_iv: "b1dc61d84f9708ba7706cc9f7b0669a4"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.352805 4763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:20.369294 4763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:20.369602 4763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:20.369725 4763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:20.369957 4763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:20.370294 4763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:20.370330 4763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:20.370368 4763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:20.370393 4763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:20.376642 4763 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:45557
I20240506 15:44:20.376699 4890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:45557 every 8 connection(s)
I20240506 15:44:20.377009 4763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:44:20.379209 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 4763
I20240506 15:44:20.379333 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:44:20.379532 18371 external_mini_cluster.cc:1428] Setting key d1229ad109bcb04238312521f88d229c
I20240506 15:44:20.380759 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33545
--builtin_ntp_servers=127.17.240.212:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240506 15:44:20.382216 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.377896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:50433 (local address 127.17.240.254:33545)
0506 15:44:20.377966 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.377968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.378018 (+ 50us) server_negotiation.cc:407] Connection header received
0506 15:44:20.378587 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.378589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.378600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.378664 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:20.379223 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.379972 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.380655 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.380870 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.381815 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:20.381819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:20.381820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:20.381825 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:20.381845 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:20.381849 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:20.381915 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.382050 (+ 135us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.382100 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240506 15:44:20.382258 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.377707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:33545 (local address 127.17.240.194:50433)
0506 15:44:20.377921 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:20.377948 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:20.378445 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:20.378729 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:20.378733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:20.378741 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:20.379096 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.379100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.380081 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.380083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:20.380493 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.380495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.380640 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.381651 (+ 1011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:44:20.381652 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:20.381674 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:44:20.381694 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:20.381946 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:20.381952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:20.381955 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:44:20.382040 (+ 85us) client_negotiation.cc:240] Negotiation successful
0506 15:44:20.382113 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20240506 15:44:20.382977 4891 heartbeater.cc:347] Connected to a master server at 127.17.240.254:33545
I20240506 15:44:20.383109 4891 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:20.383294 4891 heartbeater.cc:511] Master 127.17.240.254:33545 requested a full tablet report, sending...
I20240506 15:44:20.383733 4571 ts_manager.cc:194] Registered new tserver with Master: b8da924aa99343a283b4cd9b6eb3c31c (127.17.240.194:45557)
I20240506 15:44:20.384140 4571 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.194:50433
W20240506 15:44:20.463248 4894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:20.463483 4894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:20.463506 4894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:20.464906 4894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:20.464952 4894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:20.464977 4894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:44:20.466487 4894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:20.466701 4894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:20.466959 4894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:20.471653 4899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:20.472327 4900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:20.472950 4902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:20.473636 4894 server_base.cc:1034] running on GCE node
I20240506 15:44:20.473836 4894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:20.474042 4894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:20.475188 4894 hybrid_clock.cc:648] HybridClock initialized: now 1715010260475166 us; error 34 us; skew 500 ppm
I20240506 15:44:20.476577 4894 webserver.cc:460] Webserver started at http://127.17.240.195:39911/ using document root <none> and password file <none>
I20240506 15:44:20.476774 4894 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:20.476821 4894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:20.476961 4894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:20.477982 4894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "590642665b2f4756b0f52cc56d29d71d"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "025a6ae91a507eeaa4f8d4071f733d15"
server_key_iv: "e265337148e258ba94f969a9c1397ddc"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.478287 4894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "590642665b2f4756b0f52cc56d29d71d"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "025a6ae91a507eeaa4f8d4071f733d15"
server_key_iv: "e265337148e258ba94f969a9c1397ddc"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.479524 4894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:44:20.480252 4908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:20.480448 4894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:20.480531 4894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "590642665b2f4756b0f52cc56d29d71d"
format_stamp: "Formatted at 2024-05-06 15:44:20 on dist-test-slave-v0k1"
server_key: "025a6ae91a507eeaa4f8d4071f733d15"
server_key_iv: "e265337148e258ba94f969a9c1397ddc"
server_key_version: "encryptionkey@0"
I20240506 15:44:20.480594 4894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:20.495697 4894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:20.495966 4894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:20.496081 4894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:20.496291 4894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:20.496613 4894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:20.496650 4894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:20.496686 4894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:20.496711 4894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:20.502012 4894 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:36919
I20240506 15:44:20.502084 5021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:36919 every 8 connection(s)
I20240506 15:44:20.502360 4894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:44:20.506569 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 4894
I20240506 15:44:20.506606 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.503080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:37415 (local address 127.17.240.254:33545)
0506 15:44:20.503137 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.503139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.503340 (+ 201us) server_negotiation.cc:407] Connection header received
0506 15:44:20.503785 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.503786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.503795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.503855 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:20.504277 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.504765 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.505250 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.505358 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.506341 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:20.506345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:20.506346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:20.506349 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:20.506360 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:20.506363 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:20.506406 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.506519 (+ 113us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.506545 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240506 15:44:20.506682 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:44:20.506825 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.503071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:33545 (local address 127.17.240.195:37415)
0506 15:44:20.503231 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:20.503253 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:20.503701 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:20.503907 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:20.503911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:20.503919 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:20.504216 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.504218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.504829 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.504830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:20.505189 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.505190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.505289 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.506232 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:44:20.506232 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:20.506250 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:44:20.506269 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:20.506424 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:20.506428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:20.506430 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:44:20.506544 (+ 114us) client_negotiation.cc:240] Negotiation successful
0506 15:44:20.506628 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20240506 15:44:20.506882 18371 external_mini_cluster.cc:1428] Setting key 287040c3307a54c08ed2fe2d3559173f
I20240506 15:44:20.507308 5022 heartbeater.cc:347] Connected to a master server at 127.17.240.254:33545
I20240506 15:44:20.507412 5022 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:20.507645 5022 heartbeater.cc:511] Master 127.17.240.254:33545 requested a full tablet report, sending...
I20240506 15:44:20.508028 4571 ts_manager.cc:194] Registered new tserver with Master: 590642665b2f4756b0f52cc56d29d71d (127.17.240.195:36919)
I20240506 15:44:20.508391 4571 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.195:37415
I20240506 15:44:20.518568 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:44:20.522554 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.519919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.17.240.254:33545)
0506 15:44:20.519981 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.519983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.520063 (+ 80us) server_negotiation.cc:407] Connection header received
0506 15:44:20.520130 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.520131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.520143 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.520193 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:20.520516 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.520989 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.521538 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.521722 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.522187 (+ 465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:20.522190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:20.522191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:20.522196 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:20.522212 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:20.522216 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:20.522328 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.522440 (+ 112us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.522457 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240506 15:44:20.524261 4571 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:44:20.525676 4571 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:44:20.530735 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.527667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:45557 (local address 127.0.0.1:51692)
0506 15:44:20.528086 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:20.528094 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:20.528149 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:20.528372 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:20.528373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:20.528375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:20.528604 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.528606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.529560 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.529561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:20.530463 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.530465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.530526 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.530556 (+ 30us) client_negotiation.cc:769] Sending connection context
0506 15:44:20.530604 (+ 48us) client_negotiation.cc:240] Negotiation successful
0506 15:44:20.530648 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":4,"thread_start_us":74,"threads_started":1}
I20240506 15:44:20.530879 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.527667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:36919 (local address 127.0.0.1:51062)
0506 15:44:20.527866 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:20.527894 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:20.527967 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:20.528268 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:20.528271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:20.528274 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:20.528481 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.528483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.529817 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.529818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:20.530619 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.530621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.530673 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.530675 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:44:20.530705 (+ 30us) client_negotiation.cc:240] Negotiation successful
0506 15:44:20.530733 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20240506 15:44:20.530982 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.527671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:42489 (local address 127.0.0.1:48256)
0506 15:44:20.528205 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:20.528212 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:20.528278 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:20.528596 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:20.528597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:20.528599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:20.528765 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.528766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.530003 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.530004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:20.530753 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.530756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.530855 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.530861 (+ 6us) client_negotiation.cc:769] Sending connection context
0506 15:44:20.530895 (+ 34us) client_negotiation.cc:240] Negotiation successful
0506 15:44:20.530935 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":41,"thread_start_us":82,"threads_started":1}
I20240506 15:44:20.531606 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.527828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.17.240.195:36919)
0506 15:44:20.528092 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.528099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.528109 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:44:20.528144 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.528146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.528162 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.528245 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:20.528571 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.529738 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.530774 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.531383 (+ 609us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.531467 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.531498 (+ 31us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.531533 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":79,"threads_started":1}
I20240506 15:44:20.531653 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.527872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.17.240.193:42489)
0506 15:44:20.528031 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.528039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.528254 (+ 215us) server_negotiation.cc:407] Connection header received
0506 15:44:20.528398 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.528400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.528419 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.528522 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:20.528844 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.529931 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.531043 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.531494 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.531550 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.531575 (+ 25us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.531600 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20240506 15:44:20.532119 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.527924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.17.240.194:45557)
0506 15:44:20.528088 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.528097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.528139 (+ 42us) server_negotiation.cc:407] Connection header received
0506 15:44:20.528203 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.528204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.528221 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.528289 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:20.528688 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.529487 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.531413 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.531911 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.531980 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.532010 (+ 30us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.532043 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20240506 15:44:20.532166 4956 tablet_service.cc:1467] Processing CreateTablet for tablet 88fef37bac2045b9a22b253d00f70fb4 (DEFAULT_TABLE table=test-table [id=c43089b3af4a4853b9aebc51258620e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:20.532265 4694 tablet_service.cc:1467] Processing CreateTablet for tablet 88fef37bac2045b9a22b253d00f70fb4 (DEFAULT_TABLE table=test-table [id=c43089b3af4a4853b9aebc51258620e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:20.532491 4956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88fef37bac2045b9a22b253d00f70fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:20.532505 4694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88fef37bac2045b9a22b253d00f70fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:20.532625 4825 tablet_service.cc:1467] Processing CreateTablet for tablet 88fef37bac2045b9a22b253d00f70fb4 (DEFAULT_TABLE table=test-table [id=c43089b3af4a4853b9aebc51258620e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:20.532884 4825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88fef37bac2045b9a22b253d00f70fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:20.534754 5036 tablet_bootstrap.cc:492] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82: Bootstrap starting.
I20240506 15:44:20.535292 5036 tablet_bootstrap.cc:654] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:20.535591 5036 log.cc:824] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:20.535975 5038 tablet_bootstrap.cc:492] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: Bootstrap starting.
I20240506 15:44:20.536285 5036 tablet_bootstrap.cc:492] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82: No bootstrap required, opened a new log
I20240506 15:44:20.536350 5036 ts_tablet_manager.cc:1367] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:20.536489 5038 tablet_bootstrap.cc:654] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:20.536732 5038 log.cc:824] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:20.537022 5037 tablet_bootstrap.cc:492] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: Bootstrap starting.
I20240506 15:44:20.537348 5038 tablet_bootstrap.cc:492] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: No bootstrap required, opened a new log
I20240506 15:44:20.537400 5038 ts_tablet_manager.cc:1367] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:20.538028 5036 raft_consensus.cc:348] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 } } peers { permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 } } peers { permanent_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45557 } }
I20240506 15:44:20.538142 5036 raft_consensus.cc:374] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:20.538172 5036 raft_consensus.cc:729] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9354a260d4f477aa5fe7427654c8b82, State: Initialized, Role: FOLLOWER
I20240506 15:44:20.538264 5036 consensus_queue.cc:260] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 } } peers { permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 } } peers { permanent_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45557 } }
I20240506 15:44:20.538504 5036 ts_tablet_manager.cc:1398] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:44:20.538563 4760 heartbeater.cc:503] Master 127.17.240.254:33545 was elected leader, sending a full tablet report...
I20240506 15:44:20.539340 5038 raft_consensus.cc:348] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 } } peers { permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 } } peers { permanent_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45557 } }
I20240506 15:44:20.539475 5038 raft_consensus.cc:374] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:20.539502 5038 raft_consensus.cc:729] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8da924aa99343a283b4cd9b6eb3c31c, State: Initialized, Role: FOLLOWER
I20240506 15:44:20.539610 5038 consensus_queue.cc:260] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 } } peers { permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 } } peers { permanent_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45557 } }
I20240506 15:44:20.539881 5038 ts_tablet_manager.cc:1398] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240506 15:44:20.539929 4891 heartbeater.cc:503] Master 127.17.240.254:33545 was elected leader, sending a full tablet report...
I20240506 15:44:20.540443 5037 tablet_bootstrap.cc:654] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:20.540724 5037 log.cc:824] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:20.541373 5037 tablet_bootstrap.cc:492] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: No bootstrap required, opened a new log
I20240506 15:44:20.541431 5037 ts_tablet_manager.cc:1367] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240506 15:44:20.543133 5037 raft_consensus.cc:348] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 } } peers { permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 } } peers { permanent_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45557 } }
I20240506 15:44:20.543246 5037 raft_consensus.cc:374] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:20.543269 5037 raft_consensus.cc:729] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590642665b2f4756b0f52cc56d29d71d, State: Initialized, Role: FOLLOWER
I20240506 15:44:20.543354 5037 consensus_queue.cc:260] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 } } peers { permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 } } peers { permanent_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45557 } }
I20240506 15:44:20.543561 5037 ts_tablet_manager.cc:1398] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:44:20.543624 5022 heartbeater.cc:503] Master 127.17.240.254:33545 was elected leader, sending a full tablet report...
I20240506 15:44:20.590204 5042 raft_consensus.cc:482] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:20.590332 5042 raft_consensus.cc:504] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 } } peers { permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 } } peers { permanent_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45557 } }
I20240506 15:44:20.590823 5042 leader_election.cc:288] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 590642665b2f4756b0f52cc56d29d71d (127.17.240.195:36919), d9354a260d4f477aa5fe7427654c8b82 (127.17.240.193:42489)
I20240506 15:44:20.593915 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.590968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:36919 (local address 127.17.240.194:33143)
0506 15:44:20.591174 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:20.591185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:20.591242 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:20.591504 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:20.591505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:20.591507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:20.591718 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.591720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.592726 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.592728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:20.593658 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.593659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.593727 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.593733 (+ 6us) client_negotiation.cc:769] Sending connection context
0506 15:44:20.593771 (+ 38us) client_negotiation.cc:240] Negotiation successful
0506 15:44:20.593814 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20240506 15:44:20.593919 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.590974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:42489 (local address 127.17.240.194:56675)
0506 15:44:20.591197 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:20.591209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:20.591269 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:20.591441 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:20.591442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:20.591445 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:20.591735 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.591736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.592735 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.592735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:20.593640 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:20.593643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.593744 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.593746 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:44:20.593781 (+ 35us) client_negotiation.cc:240] Negotiation successful
0506 15:44:20.593816 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":8,"thread_start_us":143,"threads_started":1}
I20240506 15:44:20.594344 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.591261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:56675 (local address 127.17.240.193:42489)
0506 15:44:20.591325 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.591328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.591335 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:20.591362 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.591363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.591372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.591429 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:20.591818 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.592608 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.593740 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.594216 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.594223 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.594248 (+ 25us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.594277 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240506 15:44:20.594374 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.591048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:33143 (local address 127.17.240.195:36919)
0506 15:44:20.591122 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.591125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.591248 (+ 123us) server_negotiation.cc:407] Connection header received
0506 15:44:20.591418 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.591420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.591430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.591486 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:20.591821 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.592623 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.593740 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.594215 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.594223 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.594247 (+ 24us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.594277 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240506 15:44:20.594671 4976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fef37bac2045b9a22b253d00f70fb4" candidate_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590642665b2f4756b0f52cc56d29d71d" is_pre_election: true
I20240506 15:44:20.594677 4714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fef37bac2045b9a22b253d00f70fb4" candidate_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9354a260d4f477aa5fe7427654c8b82" is_pre_election: true
I20240506 15:44:20.594805 4714 raft_consensus.cc:2454] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8da924aa99343a283b4cd9b6eb3c31c in term 0.
I20240506 15:44:20.594821 4976 raft_consensus.cc:2454] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8da924aa99343a283b4cd9b6eb3c31c in term 0.
I20240506 15:44:20.595085 4781 leader_election.cc:302] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 590642665b2f4756b0f52cc56d29d71d, b8da924aa99343a283b4cd9b6eb3c31c; no voters:
I20240506 15:44:20.595206 5042 raft_consensus.cc:2781] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:44:20.595278 5042 raft_consensus.cc:482] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:44:20.595311 5042 raft_consensus.cc:3037] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:20.595999 5042 raft_consensus.cc:504] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 } } peers { permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 } } peers { permanent_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45557 } }
I20240506 15:44:20.596138 5042 leader_election.cc:288] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [CANDIDATE]: Term 1 election: Requested vote from peers 590642665b2f4756b0f52cc56d29d71d (127.17.240.195:36919), d9354a260d4f477aa5fe7427654c8b82 (127.17.240.193:42489)
I20240506 15:44:20.596323 4976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fef37bac2045b9a22b253d00f70fb4" candidate_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590642665b2f4756b0f52cc56d29d71d"
I20240506 15:44:20.596376 4714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fef37bac2045b9a22b253d00f70fb4" candidate_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9354a260d4f477aa5fe7427654c8b82"
I20240506 15:44:20.596407 4976 raft_consensus.cc:3037] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:20.596441 4714 raft_consensus.cc:3037] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:20.597108 4976 raft_consensus.cc:2454] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8da924aa99343a283b4cd9b6eb3c31c in term 1.
I20240506 15:44:20.597129 4714 raft_consensus.cc:2454] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8da924aa99343a283b4cd9b6eb3c31c in term 1.
I20240506 15:44:20.597276 4781 leader_election.cc:302] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 590642665b2f4756b0f52cc56d29d71d, b8da924aa99343a283b4cd9b6eb3c31c; no voters:
I20240506 15:44:20.597393 5042 raft_consensus.cc:2781] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:20.597558 5042 raft_consensus.cc:686] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 1 LEADER]: Becoming Leader. State: Replica: b8da924aa99343a283b4cd9b6eb3c31c, State: Running, Role: LEADER
I20240506 15:44:20.597653 5042 consensus_queue.cc:237] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 } } peers { permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 } } peers { permanent_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45557 } }
I20240506 15:44:20.598476 4571 catalog_manager.cc:5484] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c reported cstate change: term changed from 0 to 1, leader changed from <none> to b8da924aa99343a283b4cd9b6eb3c31c (127.17.240.194). New cstate: current_term: 1 leader_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8da924aa99343a283b4cd9b6eb3c31c" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45557 } health_report { overall_health: HEALTHY } } }
I20240506 15:44:20.608201 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.605961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.17.240.194:45557)
0506 15:44:20.606016 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:20.606018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:20.606027 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:44:20.606120 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:20.606128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:20.606145 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:20.606207 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:20.606480 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.606962 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:20.607559 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:20.607713 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:20.607870 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:20.607956 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:20.608039 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:20.608119 (+ 80us) server_negotiation.cc:299] Negotiation successful
0506 15:44:20.608130 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240506 15:44:20.609867 4976 raft_consensus.cc:1261] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d [term 1 FOLLOWER]: Refusing update from remote peer b8da924aa99343a283b4cd9b6eb3c31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:20.609896 4714 raft_consensus.cc:1261] T 88fef37bac2045b9a22b253d00f70fb4 P d9354a260d4f477aa5fe7427654c8b82 [term 1 FOLLOWER]: Refusing update from remote peer b8da924aa99343a283b4cd9b6eb3c31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:20.610199 5046 consensus_queue.cc:1035] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "590642665b2f4756b0f52cc56d29d71d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:20.610320 5042 consensus_queue.cc:1035] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9354a260d4f477aa5fe7427654c8b82" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 42489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:20.612926 5050 mvcc.cc:205] Tried to move back new op lower bound from 7024682027455467520 to 7024682027408633856. Current Snapshot: MvccSnapshot[applied={T|T < 7024682027455467520}]
I20240506 15:44:20.613967 5051 mvcc.cc:205] Tried to move back new op lower bound from 7024682027455467520 to 7024682027408633856. Current Snapshot: MvccSnapshot[applied={T|T < 7024682027455467520}]
I20240506 15:44:20.615159 4571 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33070:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:20.615255 4571 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33070:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:20.615975 4571 catalog_manager.cc:5771] T 00000000000000000000000000000000 P da241d71c29d40a09d6db164d334b0bb: Sending DeleteTablet for 3 replicas of tablet 88fef37bac2045b9a22b253d00f70fb4
I20240506 15:44:20.616277 4825 tablet_service.cc:1514] Processing DeleteTablet for tablet 88fef37bac2045b9a22b253d00f70fb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:20 UTC) from {username='slave'} at 127.0.0.1:51692
I20240506 15:44:20.616277 4956 tablet_service.cc:1514] Processing DeleteTablet for tablet 88fef37bac2045b9a22b253d00f70fb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:20 UTC) from {username='slave'} at 127.0.0.1:51062
I20240506 15:44:20.616408 4694 tablet_service.cc:1514] Processing DeleteTablet for tablet 88fef37bac2045b9a22b253d00f70fb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:20 UTC) from {username='slave'} at 127.0.0.1:48256
I20240506 15:44:20.616508 5058 tablet_replica.cc:330] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: stopping tablet replica
I20240506 15:44:20.616586 5058 raft_consensus.cc:2229] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:44:20.616681 5058 raft_consensus.cc:2258] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:20.616824 5059 tablet_replica.cc:330] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: stopping tablet replica
I20240506 15:44:20.616855 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 4632
I20240506 15:44:20.616905 5059 raft_consensus.cc:2229] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:44:20.616991 5059 raft_consensus.cc:2258] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:20.617372 5058 ts_tablet_manager.cc:1875] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:44:20.617893 5059 ts_tablet_manager.cc:1875] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:44:20.619367 5058 ts_tablet_manager.cc:1888] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:44:20.619441 5058 log.cc:1196] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/88fef37bac2045b9a22b253d00f70fb4
I20240506 15:44:20.619719 5058 ts_tablet_manager.cc:1909] T 88fef37bac2045b9a22b253d00f70fb4 P 590642665b2f4756b0f52cc56d29d71d: Deleting consensus metadata
I20240506 15:44:20.619946 5059 ts_tablet_manager.cc:1888] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:44:20.620064 5059 log.cc:1196] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/88fef37bac2045b9a22b253d00f70fb4
I20240506 15:44:20.620131 4559 catalog_manager.cc:4830] TS 590642665b2f4756b0f52cc56d29d71d (127.17.240.195:36919): tablet 88fef37bac2045b9a22b253d00f70fb4 (table test-table [id=c43089b3af4a4853b9aebc51258620e9]) successfully deleted
I20240506 15:44:20.620432 5059 ts_tablet_manager.cc:1909] T 88fef37bac2045b9a22b253d00f70fb4 P b8da924aa99343a283b4cd9b6eb3c31c: Deleting consensus metadata
I20240506 15:44:20.620751 4557 catalog_manager.cc:4830] TS b8da924aa99343a283b4cd9b6eb3c31c (127.17.240.194:45557): tablet 88fef37bac2045b9a22b253d00f70fb4 (table test-table [id=c43089b3af4a4853b9aebc51258620e9]) successfully deleted
W20240506 15:44:20.625924 4556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:42489 (error 108)
I20240506 15:44:20.626396 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:20.626271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:42489 (local address 127.0.0.1:48262)
0506 15:44:20.626322 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:20.626361 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:42489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20240506 15:44:20.626483 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 4763
W20240506 15:44:20.626591 4556 catalog_manager.cc:4573] TS d9354a260d4f477aa5fe7427654c8b82 (127.17.240.193:42489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88fef37bac2045b9a22b253d00f70fb4: Network error: Client connection negotiation failed: client connection to 127.17.240.193:42489: connect: Connection refused (error 111)
I20240506 15:44:20.632167 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 4894
I20240506 15:44:20.636843 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 4541
2024-05-06T15:44:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (700 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:20 dist-test-slave-v0k1 krb5kdc[5064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:20 dist-test-slave-v0k1 krb5kdc[5064](info): set up 2 sockets
May 06 15:44:20 dist-test-slave-v0k1 krb5kdc[5064](info): commencing operation
krb5kdc: starting...
W20240506 15:44:21.599117 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:21 dist-test-slave-v0k1 krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:21.711776 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38589
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:21.791705 5077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:21.791909 5077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:21.791931 5077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:21.793241 5077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:21.793282 5077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:21.793296 5077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:21.793308 5077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:21.793319 5077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:21.794889 5077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:38589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:21.795121 5077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:21.795274 5077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:21.800220 5083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:21.800693 5085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:21.801430 5082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:21.802130 5077 server_base.cc:1034] running on GCE node
I20240506 15:44:21.802361 5077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:21.802623 5077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:21.803786 5077 hybrid_clock.cc:648] HybridClock initialized: now 1715010261803768 us; error 34 us; skew 500 ppm
May 06 15:44:21 dist-test-slave-v0k1 krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010261, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:21.811337 5077 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:21.811802 5077 webserver.cc:460] Webserver started at http://127.17.240.254:35753/ using document root <none> and password file <none>
I20240506 15:44:21.812009 5077 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:21.812053 5077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:21.812175 5077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:21.813364 5077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "33b3f69494ed4d5198a46bfc3e9b897a"
format_stamp: "Formatted at 2024-05-06 15:44:21 on dist-test-slave-v0k1"
server_key: "a260568e54c75c0a87e735f8c51ef9b2"
server_key_iv: "48353e507efb053a4c9c3a379c647ccb"
server_key_version: "encryptionkey@0"
I20240506 15:44:21.813777 5077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "33b3f69494ed4d5198a46bfc3e9b897a"
format_stamp: "Formatted at 2024-05-06 15:44:21 on dist-test-slave-v0k1"
server_key: "a260568e54c75c0a87e735f8c51ef9b2"
server_key_iv: "48353e507efb053a4c9c3a379c647ccb"
server_key_version: "encryptionkey@0"
I20240506 15:44:21.815105 5077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:44:21.815826 5092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:21.816025 5077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:21.816107 5077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "33b3f69494ed4d5198a46bfc3e9b897a"
format_stamp: "Formatted at 2024-05-06 15:44:21 on dist-test-slave-v0k1"
server_key: "a260568e54c75c0a87e735f8c51ef9b2"
server_key_iv: "48353e507efb053a4c9c3a379c647ccb"
server_key_version: "encryptionkey@0"
I20240506 15:44:21.816182 5077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:21.828114 5077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:21.830260 5077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:21.830374 5077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:21.834160 5077 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:38589
I20240506 15:44:21.834190 5144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:38589 every 8 connection(s)
I20240506 15:44:21.834503 5077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:21.835341 5145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:21.836411 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 5077
I20240506 15:44:21.836520 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:21.836673 18371 external_mini_cluster.cc:1428] Setting key 884a7ca47eed7620adcd1fd2ef34d398
I20240506 15:44:21.837826 5145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a: Bootstrap starting.
I20240506 15:44:21.838485 5145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:21.838737 5145 log.cc:824] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:21.839380 5145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a: No bootstrap required, opened a new log
I20240506 15:44:21.841073 5145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3f69494ed4d5198a46bfc3e9b897a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38589 } }
I20240506 15:44:21.841194 5145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:21.841225 5145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b3f69494ed4d5198a46bfc3e9b897a, State: Initialized, Role: FOLLOWER
I20240506 15:44:21.841348 5145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3f69494ed4d5198a46bfc3e9b897a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38589 } }
I20240506 15:44:21.841426 5145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:21.841467 5145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:21.841512 5145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:21.842031 5145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3f69494ed4d5198a46bfc3e9b897a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38589 } }
I20240506 15:44:21.842128 5145 leader_election.cc:302] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33b3f69494ed4d5198a46bfc3e9b897a; no voters:
I20240506 15:44:21.842273 5145 leader_election.cc:288] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:21.842319 5150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [term 1 FOLLOWER]: Leader election won for term 1
May 06 15:44:21 dist-test-slave-v0k1 krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:21.842455 5150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [term 1 LEADER]: Becoming Leader. State: Replica: 33b3f69494ed4d5198a46bfc3e9b897a, State: Running, Role: LEADER
I20240506 15:44:21.842521 5145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:21.842536 5150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3f69494ed4d5198a46bfc3e9b897a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38589 } }
I20240506 15:44:21.842828 5150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33b3f69494ed4d5198a46bfc3e9b897a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3f69494ed4d5198a46bfc3e9b897a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38589 } } }
I20240506 15:44:21.842893 5150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:21.843158 5150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33b3f69494ed4d5198a46bfc3e9b897a. Latest consensus state: current_term: 1 leader_uuid: "33b3f69494ed4d5198a46bfc3e9b897a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b3f69494ed4d5198a46bfc3e9b897a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 38589 } } }
I20240506 15:44:21.843236 5150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:21.843640 5154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:21.844393 5154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:21.845657 5154 catalog_manager.cc:1347] Generated new cluster ID: 88b8a4212fd14d998e12f8a16cda8223
I20240506 15:44:21.845711 5154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:21.847806 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:21.837178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59088 (local address 127.17.240.254:38589)
0506 15:44:21.837389 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:21.837400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:21.837431 (+ 31us) server_negotiation.cc:407] Connection header received
0506 15:44:21.838017 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:21.838025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:21.838080 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:21.838253 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:21.838999 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:21.839818 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:21.840392 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:21.840634 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:21.842963 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:21.842981 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:21.842982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:21.843001 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:21.845556 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:21.846713 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:21.846716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:21.846716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:21.846778 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:21.846972 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:21.846973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:21.846973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:21.847231 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:21.847392 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:21.847484 (+ 92us) server_negotiation.cc:299] Negotiation successful
0506 15:44:21.847610 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":112,"threads_started":1}
I20240506 15:44:21.852855 5154 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:21.853344 5154 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:21.859062 5154 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a: Generated new TSK 0
I20240506 15:44:21.859236 5154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
Entry for principal kudu/127.17.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab.
I20240506 15:44:21.889950 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38589
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:21.970436 5171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:21.970634 5171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:21.970667 5171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:21.972067 5171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:21.972115 5171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:21.972153 5171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:44:21.973716 5171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.193.keytab
--principal=kudu/127.17.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:21.973948 5171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:21.974143 5171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:21.979463 5177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:21.980036 5176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:21.980554 5179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:21.981388 5171 server_base.cc:1034] running on GCE node
I20240506 15:44:21.981587 5171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:21.981809 5171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:21.982961 5171 hybrid_clock.cc:648] HybridClock initialized: now 1715010261982943 us; error 41 us; skew 500 ppm
May 06 15:44:21 dist-test-slave-v0k1 krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010261, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:21.985689 5171 init.cc:377] Logged in from keytab as kudu/127.17.240.193@KRBTEST.COM (short username kudu)
I20240506 15:44:21.986105 5171 webserver.cc:460] Webserver started at http://127.17.240.193:37989/ using document root <none> and password file <none>
I20240506 15:44:21.986284 5171 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:21.986330 5171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:21.986454 5171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:21.987541 5171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "1d774642ccb343338152ff1efd12d036"
format_stamp: "Formatted at 2024-05-06 15:44:21 on dist-test-slave-v0k1"
server_key: "371677692d05ebafe2aa402388d48a97"
server_key_iv: "d75d4aebd1b3342e1dda3930821943e5"
server_key_version: "encryptionkey@0"
I20240506 15:44:21.987854 5171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "1d774642ccb343338152ff1efd12d036"
format_stamp: "Formatted at 2024-05-06 15:44:21 on dist-test-slave-v0k1"
server_key: "371677692d05ebafe2aa402388d48a97"
server_key_iv: "d75d4aebd1b3342e1dda3930821943e5"
server_key_version: "encryptionkey@0"
I20240506 15:44:21.989101 5171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:44:21.989814 5186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:21.990006 5171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:21.990095 5171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "1d774642ccb343338152ff1efd12d036"
format_stamp: "Formatted at 2024-05-06 15:44:21 on dist-test-slave-v0k1"
server_key: "371677692d05ebafe2aa402388d48a97"
server_key_iv: "d75d4aebd1b3342e1dda3930821943e5"
server_key_version: "encryptionkey@0"
I20240506 15:44:21.990159 5171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:22.003604 5171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:22.005802 5171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:22.005935 5171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:22.006168 5171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:22.006494 5171 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:22.006529 5171 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.006565 5171 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:22.006603 5171 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.012097 5171 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:34875
I20240506 15:44:22.012131 5299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:34875 every 8 connection(s)
I20240506 15:44:22.012434 5171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:44:22.014731 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 5171
I20240506 15:44:22.014868 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:44:22.015100 18371 external_mini_cluster.cc:1428] Setting key 1d3c5d43072fc185c8806a09a2fea0bd
May 06 15:44:22 dist-test-slave-v0k1 krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010261, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.193@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:22.022039 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.013133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38589 (local address 127.17.240.193:56047)
0506 15:44:22.013284 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.013314 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:22.013756 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:22.013971 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:22.013974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:22.014274 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:22.015104 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.015112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.015955 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.015956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:22.016562 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.016566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.016717 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.017293 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:22.017320 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:22.018718 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:22.020818 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:22.020826 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:22.020829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:22.021080 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:22.021244 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:22.021245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:22.021245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:22.021298 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:22.021557 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:22.021563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:22.021685 (+ 122us) client_negotiation.cc:769] Sending connection context
0506 15:44:22.021779 (+ 94us) client_negotiation.cc:240] Negotiation successful
0506 15:44:22.021889 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20240506 15:44:22.022044 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.013169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:56047 (local address 127.17.240.254:38589)
0506 15:44:22.013226 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.013228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.013399 (+ 171us) server_negotiation.cc:407] Connection header received
0506 15:44:22.013845 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.013846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.013862 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.013924 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:22.015239 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.015870 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.016676 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.016818 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.018848 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:22.018852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:22.018853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:22.018861 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:22.020728 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:22.021152 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:22.021153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:22.021154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:22.021183 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:22.021350 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:22.021350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:22.021351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:22.021484 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:22.021550 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.021818 (+ 268us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.021926 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240506 15:44:22.022809 5300 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38589
I20240506 15:44:22.022923 5300 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:22.023171 5300 heartbeater.cc:511] Master 127.17.240.254:38589 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.240.194@KRBTEST.COM; defaulting to no policy
I20240506 15:44:22.023757 5109 ts_manager.cc:194] Registered new tserver with Master: 1d774642ccb343338152ff1efd12d036 (127.17.240.193:34875)
I20240506 15:44:22.024410 5109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.193@KRBTEST.COM'} at 127.17.240.193:56047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
Entry for principal kudu/127.17.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab.
I20240506 15:44:22.042287 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38589
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:22.120931 5305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:22.121140 5305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:22.121166 5305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:22.122467 5305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:22.122509 5305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:22.122534 5305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:44:22.123999 5305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.194.keytab
--principal=kudu/127.17.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:22.124187 5305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:22.124374 5305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:22.128840 5313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:22.129518 5311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:22.130089 5310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:22.130683 5305 server_base.cc:1034] running on GCE node
I20240506 15:44:22.130841 5305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:22.131068 5305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:22.132197 5305 hybrid_clock.cc:648] HybridClock initialized: now 1715010262132177 us; error 37 us; skew 500 ppm
May 06 15:44:22 dist-test-slave-v0k1 krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010262, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:22.134593 5305 init.cc:377] Logged in from keytab as kudu/127.17.240.194@KRBTEST.COM (short username kudu)
I20240506 15:44:22.134974 5305 webserver.cc:460] Webserver started at http://127.17.240.194:43035/ using document root <none> and password file <none>
I20240506 15:44:22.135177 5305 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:22.135221 5305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:22.135327 5305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:22.136309 5305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "c1ce20ee78b041d1a88330291c2e0b42"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "6d498f2a472810129d89e538962bdbd0"
server_key_iv: "8a8673c5474cff9f0553d54692d86198"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.136618 5305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "c1ce20ee78b041d1a88330291c2e0b42"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "6d498f2a472810129d89e538962bdbd0"
server_key_iv: "8a8673c5474cff9f0553d54692d86198"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.137770 5305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:44:22.138438 5320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.138680 5305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:44:22.138768 5305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "c1ce20ee78b041d1a88330291c2e0b42"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "6d498f2a472810129d89e538962bdbd0"
server_key_iv: "8a8673c5474cff9f0553d54692d86198"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.138829 5305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:22.147397 5305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:22.149375 5305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:22.149490 5305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:22.149677 5305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:22.149958 5305 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:22.149998 5305 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.150031 5305 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:22.150060 5305 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.155414 5305 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:45861
I20240506 15:44:22.155479 5433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:45861 every 8 connection(s)
I20240506 15:44:22.155745 5305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:44:22.156538 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 5305
I20240506 15:44:22.156666 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:44:22.156903 18371 external_mini_cluster.cc:1428] Setting key 4763a5006d023a38b7a3cf12bc01f1fa
May 06 15:44:22 dist-test-slave-v0k1 krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010262, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.194@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.240.195@KRBTEST.COM; defaulting to no policy
I20240506 15:44:22.164568 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.156439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:55845 (local address 127.17.240.254:38589)
0506 15:44:22.156515 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.156518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.156745 (+ 227us) server_negotiation.cc:407] Connection header received
0506 15:44:22.157310 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.157311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.157320 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.157359 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:22.158711 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.159242 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.159788 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.159961 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.162081 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:22.162085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:22.162086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:22.162093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:22.163468 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:22.163825 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:22.163826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:22.163826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:22.163849 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:22.164012 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:22.164013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:22.164013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:22.164101 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:22.164159 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.164391 (+ 232us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.164477 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240506 15:44:22.164700 5436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.156395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38589 (local address 127.17.240.194:55845)
0506 15:44:22.156575 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.156610 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:22.157205 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:22.157414 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:22.157418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:22.157799 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:22.158613 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.158621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.159335 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.159336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:22.159707 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.159709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.159841 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.160406 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:22.160421 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:22.161992 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:22.163549 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:22.163554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:22.163557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:22.163759 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:22.163901 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:22.163902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:22.163903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:22.163958 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:22.164154 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:22.164159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:22.164272 (+ 113us) client_negotiation.cc:769] Sending connection context
0506 15:44:22.164370 (+ 98us) client_negotiation.cc:240] Negotiation successful
0506 15:44:22.164511 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20240506 15:44:22.165175 5434 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38589
I20240506 15:44:22.165256 5434 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:22.165432 5434 heartbeater.cc:511] Master 127.17.240.254:38589 requested a full tablet report, sending...
I20240506 15:44:22.165791 5109 ts_manager.cc:194] Registered new tserver with Master: c1ce20ee78b041d1a88330291c2e0b42 (127.17.240.194:45861)
I20240506 15:44:22.166160 5109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.194@KRBTEST.COM'} at 127.17.240.194:55845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
Entry for principal kudu/127.17.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab.
I20240506 15:44:22.179556 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38589
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:22.262527 5439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:22.262737 5439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:22.262763 5439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:22.264304 5439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:22.264379 5439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:22.264408 5439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:44:22.266139 5439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.195.keytab
--principal=kudu/127.17.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:22.266394 5439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:22.266595 5439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:22.271986 5445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:22.272716 5447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:22.273509 5444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:22.274086 5439 server_base.cc:1034] running on GCE node
I20240506 15:44:22.274300 5439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:22.274546 5439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:22.275688 5439 hybrid_clock.cc:648] HybridClock initialized: now 1715010262275674 us; error 39 us; skew 500 ppm
May 06 15:44:22 dist-test-slave-v0k1 krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010262, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:22.278191 5439 init.cc:377] Logged in from keytab as kudu/127.17.240.195@KRBTEST.COM (short username kudu)
I20240506 15:44:22.278635 5439 webserver.cc:460] Webserver started at http://127.17.240.195:42939/ using document root <none> and password file <none>
I20240506 15:44:22.278805 5439 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:22.278847 5439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:22.278934 5439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:22.279959 5439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "8cfd2c902d154db997d8f5c01be40a2d"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "50cb2d17dd1bea987286b61fb33046f3"
server_key_iv: "41dbb41555b1b3ffcd6fb5df86bf6f75"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.280233 5439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "8cfd2c902d154db997d8f5c01be40a2d"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "50cb2d17dd1bea987286b61fb33046f3"
server_key_iv: "41dbb41555b1b3ffcd6fb5df86bf6f75"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.281419 5439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:44:22.282060 5454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.282222 5439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:22.282300 5439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "8cfd2c902d154db997d8f5c01be40a2d"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "50cb2d17dd1bea987286b61fb33046f3"
server_key_iv: "41dbb41555b1b3ffcd6fb5df86bf6f75"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.282369 5439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:22.324065 5439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:22.325930 5439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:22.326054 5439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:22.326272 5439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:22.326577 5439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:22.326612 5439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.326649 5439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:22.326678 5439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.332243 5439 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:33841
I20240506 15:44:22.332288 5567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:33841 every 8 connection(s)
I20240506 15:44:22.332640 5439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:44:22.334348 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 5439
I20240506 15:44:22.334468 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:44:22.334664 18371 external_mini_cluster.cc:1428] Setting key 7ae1073df731c0b258ac9c35991a6cd9
May 06 15:44:22 dist-test-slave-v0k1 krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010262, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.195@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:22.340792 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.333331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:46143 (local address 127.17.240.254:38589)
0506 15:44:22.333389 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.333390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.333561 (+ 171us) server_negotiation.cc:407] Connection header received
0506 15:44:22.334008 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.334009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.334018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.334076 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:22.335487 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.336013 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.336565 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.336664 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.338520 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:22.338524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:22.338524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:22.338530 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:22.339855 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:22.340192 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:22.340192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:22.340193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:22.340209 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:22.340344 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:22.340344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:22.340344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:22.340426 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:22.340475 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.340665 (+ 190us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.340729 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240506 15:44:22.340890 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.333295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:38589 (local address 127.17.240.195:46143)
0506 15:44:22.333454 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.333484 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:22.333927 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:22.334184 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:22.334190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:22.334685 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:22.335385 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.335391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.336092 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.336093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:22.336496 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.336498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.336607 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.337085 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:22.337098 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:22.338435 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:22.339925 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:22.339930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:22.339932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:22.340131 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:22.340254 (+ 123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:22.340255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:22.340255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:22.340301 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:22.340467 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:22.340469 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:22.340569 (+ 100us) client_negotiation.cc:769] Sending connection context
0506 15:44:22.340650 (+ 81us) client_negotiation.cc:240] Negotiation successful
0506 15:44:22.340749 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20240506 15:44:22.341282 5568 heartbeater.cc:347] Connected to a master server at 127.17.240.254:38589
I20240506 15:44:22.341378 5568 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:22.341596 5568 heartbeater.cc:511] Master 127.17.240.254:38589 requested a full tablet report, sending...
I20240506 15:44:22.342048 5109 ts_manager.cc:194] Registered new tserver with Master: 8cfd2c902d154db997d8f5c01be40a2d (127.17.240.195:33841)
I20240506 15:44:22.342391 5109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.240.195@KRBTEST.COM'} at 127.17.240.195:46143
I20240506 15:44:22.346478 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:44:22.352664 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.347843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.17.240.254:38589)
0506 15:44:22.347902 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.347904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.347995 (+ 91us) server_negotiation.cc:407] Connection header received
0506 15:44:22.348045 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.348046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.348053 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.348102 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:22.348692 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.349136 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.349658 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.349754 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.350674 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:22.350676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:22.350676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:22.350680 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:22.351901 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:22.352167 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:22.352168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:22.352168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:22.352184 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:22.352284 (+ 100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:22.352284 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:22.352284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:22.352381 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:22.352451 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.352532 (+ 81us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.352591 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:44:22.354256 5109 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:44:22.355787 5109 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:44:22.360653 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.357653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:33841 (local address 127.0.0.1:42882)
0506 15:44:22.358096 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.358105 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:22.358155 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:22.358339 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:22.358340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:22.358342 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:22.358548 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.358550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.359575 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.359576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:22.360377 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.360379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.360452 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.360488 (+ 36us) client_negotiation.cc:769] Sending connection context
0506 15:44:22.360538 (+ 50us) client_negotiation.cc:240] Negotiation successful
0506 15:44:22.360576 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":52,"threads_started":1}
I20240506 15:44:22.361073 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.357648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:45861 (local address 127.0.0.1:56536)
0506 15:44:22.357895 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.357920 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:22.357999 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:22.358654 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:22.358655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:22.358658 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:22.358853 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.358854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.360090 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.360091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:22.360849 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.360851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.360924 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.360927 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:44:22.360961 (+ 34us) client_negotiation.cc:240] Negotiation successful
0506 15:44:22.361007 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":55,"threads_started":1}
I20240506 15:44:22.361510 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.357879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.17.240.195:33841)
0506 15:44:22.358034 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.358043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.358165 (+ 122us) server_negotiation.cc:407] Connection header received
0506 15:44:22.358225 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.358227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.358243 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.358332 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:22.358633 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.359484 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.360743 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.361303 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.361354 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.361384 (+ 30us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.361424 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20240506 15:44:22.361550 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.357873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.17.240.193:34875)
0506 15:44:22.358056 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.358065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.358074 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:44:22.358117 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.358119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.358136 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.358218 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:22.358574 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.359734 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.360884 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.361403 (+ 519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.361442 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.361469 (+ 27us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.361497 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20240506 15:44:22.360816 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.357651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:34875 (local address 127.0.0.1:57138)
0506 15:44:22.357895 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.357921 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:22.358009 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:22.358223 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:22.358226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:22.358229 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:22.358465 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.358467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.359812 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.359813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:22.360624 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.360626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.360689 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.360692 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:44:22.360722 (+ 30us) client_negotiation.cc:240] Negotiation successful
0506 15:44:22.360768 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":58,"threads_started":1}
I20240506 15:44:22.362056 5502 tablet_service.cc:1467] Processing CreateTablet for tablet 32613bf8208142eda18e9aed5f921ad5 (DEFAULT_TABLE table=test-table [id=4aa4488d1d6e4f1a9ada10e849ac05ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:22.362165 5234 tablet_service.cc:1467] Processing CreateTablet for tablet 32613bf8208142eda18e9aed5f921ad5 (DEFAULT_TABLE table=test-table [id=4aa4488d1d6e4f1a9ada10e849ac05ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:22.362391 5502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32613bf8208142eda18e9aed5f921ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:22.362413 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32613bf8208142eda18e9aed5f921ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:22.365538 5582 tablet_bootstrap.cc:492] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d: Bootstrap starting.
I20240506 15:44:22.365881 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.357730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.17.240.194:45861)
0506 15:44:22.358347 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.358354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.358362 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:44:22.358401 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.358402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.358418 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.358583 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:22.358934 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.360015 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.361270 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.365646 (+ 4376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.365698 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.365733 (+ 35us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.365795 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":90,"threads_started":1}
I20240506 15:44:22.366286 5582 tablet_bootstrap.cc:654] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:22.366387 5368 tablet_service.cc:1467] Processing CreateTablet for tablet 32613bf8208142eda18e9aed5f921ad5 (DEFAULT_TABLE table=test-table [id=4aa4488d1d6e4f1a9ada10e849ac05ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:22.366552 5582 log.cc:824] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:22.366672 5368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32613bf8208142eda18e9aed5f921ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:22.366892 5583 tablet_bootstrap.cc:492] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036: Bootstrap starting.
I20240506 15:44:22.367322 5582 tablet_bootstrap.cc:492] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d: No bootstrap required, opened a new log
I20240506 15:44:22.367393 5582 ts_tablet_manager.cc:1367] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:22.367534 5583 tablet_bootstrap.cc:654] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:22.367825 5583 log.cc:824] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:22.368603 5583 tablet_bootstrap.cc:492] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036: No bootstrap required, opened a new log
I20240506 15:44:22.368674 5583 ts_tablet_manager.cc:1367] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:22.369505 5586 tablet_bootstrap.cc:492] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: Bootstrap starting.
I20240506 15:44:22.369815 5582 raft_consensus.cc:348] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 } } peers { permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 } } peers { permanent_uuid: "1d774642ccb343338152ff1efd12d036" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34875 } }
I20240506 15:44:22.369962 5582 raft_consensus.cc:374] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:22.369990 5582 raft_consensus.cc:729] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfd2c902d154db997d8f5c01be40a2d, State: Initialized, Role: FOLLOWER
I20240506 15:44:22.369987 5586 tablet_bootstrap.cc:654] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:22.370105 5582 consensus_queue.cc:260] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 } } peers { permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 } } peers { permanent_uuid: "1d774642ccb343338152ff1efd12d036" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34875 } }
I20240506 15:44:22.370270 5586 log.cc:824] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:22.370347 5582 ts_tablet_manager.cc:1398] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:22.370483 5568 heartbeater.cc:503] Master 127.17.240.254:38589 was elected leader, sending a full tablet report...
I20240506 15:44:22.370894 5586 tablet_bootstrap.cc:492] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: No bootstrap required, opened a new log
I20240506 15:44:22.370958 5586 ts_tablet_manager.cc:1367] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:22.371125 5583 raft_consensus.cc:348] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 } } peers { permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 } } peers { permanent_uuid: "1d774642ccb343338152ff1efd12d036" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34875 } }
I20240506 15:44:22.371239 5583 raft_consensus.cc:374] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:22.371268 5583 raft_consensus.cc:729] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d774642ccb343338152ff1efd12d036, State: Initialized, Role: FOLLOWER
I20240506 15:44:22.371443 5583 consensus_queue.cc:260] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 } } peers { permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 } } peers { permanent_uuid: "1d774642ccb343338152ff1efd12d036" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34875 } }
I20240506 15:44:22.371695 5583 ts_tablet_manager.cc:1398] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:22.371807 5300 heartbeater.cc:503] Master 127.17.240.254:38589 was elected leader, sending a full tablet report...
I20240506 15:44:22.372937 5586 raft_consensus.cc:348] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 } } peers { permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 } } peers { permanent_uuid: "1d774642ccb343338152ff1efd12d036" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34875 } }
I20240506 15:44:22.373044 5586 raft_consensus.cc:374] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:22.373060 5586 raft_consensus.cc:729] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ce20ee78b041d1a88330291c2e0b42, State: Initialized, Role: FOLLOWER
I20240506 15:44:22.373127 5586 consensus_queue.cc:260] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 } } peers { permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 } } peers { permanent_uuid: "1d774642ccb343338152ff1efd12d036" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34875 } }
I20240506 15:44:22.373332 5586 ts_tablet_manager.cc:1398] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 15:44:22.373400 5434 heartbeater.cc:503] Master 127.17.240.254:38589 was elected leader, sending a full tablet report...
W20240506 15:44:22.406288 5435 tablet.cc:2369] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:44:22.443152 5589 raft_consensus.cc:482] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:22.443287 5589 raft_consensus.cc:504] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 } } peers { permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 } } peers { permanent_uuid: "1d774642ccb343338152ff1efd12d036" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34875 } }
I20240506 15:44:22.443636 5589 leader_election.cc:288] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1ce20ee78b041d1a88330291c2e0b42 (127.17.240.194:45861), 8cfd2c902d154db997d8f5c01be40a2d (127.17.240.195:33841)
I20240506 15:44:22.445700 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.443807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:33841 (local address 127.17.240.193:56827)
0506 15:44:22.443861 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.443875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:22.443925 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:22.444113 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:22.444114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:22.444116 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:22.444301 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.444303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.444923 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.444925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:22.445489 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.445490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.445558 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.445562 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:44:22.445594 (+ 32us) client_negotiation.cc:240] Negotiation successful
0506 15:44:22.445627 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20240506 15:44:22.445873 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.443808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:45861 (local address 127.17.240.193:52615)
0506 15:44:22.444009 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.444020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:22.444084 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:22.444215 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:22.444215 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:22.444217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:22.444454 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.444455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.445097 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.445098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:22.445655 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.445657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.445724 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.445729 (+ 5us) client_negotiation.cc:769] Sending connection context
0506 15:44:22.445760 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:44:22.445794 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":12,"thread_start_us":122,"threads_started":1}
I20240506 15:44:22.446219 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.443959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:56827 (local address 127.17.240.195:33841)
0506 15:44:22.444009 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.444011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.444018 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:22.444041 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.444041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.444049 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.444096 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:22.444377 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.444845 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.445572 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.446098 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.446105 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.446127 (+ 22us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.446155 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240506 15:44:22.446271 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.443917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:52615 (local address 127.17.240.194:45861)
0506 15:44:22.443976 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.443978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.444084 (+ 106us) server_negotiation.cc:407] Connection header received
0506 15:44:22.444151 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.444152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.444161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.444212 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:22.444530 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.445001 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.445739 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.446175 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.446182 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.446201 (+ 19us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.446223 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240506 15:44:22.446511 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32613bf8208142eda18e9aed5f921ad5" candidate_uuid: "1d774642ccb343338152ff1efd12d036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cfd2c902d154db997d8f5c01be40a2d" is_pre_election: true
I20240506 15:44:22.446527 5388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32613bf8208142eda18e9aed5f921ad5" candidate_uuid: "1d774642ccb343338152ff1efd12d036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ce20ee78b041d1a88330291c2e0b42" is_pre_election: true
I20240506 15:44:22.446681 5388 raft_consensus.cc:2454] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d774642ccb343338152ff1efd12d036 in term 0.
I20240506 15:44:22.446673 5522 raft_consensus.cc:2454] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d774642ccb343338152ff1efd12d036 in term 0.
I20240506 15:44:22.446880 5189 leader_election.cc:302] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d774642ccb343338152ff1efd12d036, c1ce20ee78b041d1a88330291c2e0b42; no voters:
I20240506 15:44:22.447042 5589 raft_consensus.cc:2781] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:44:22.447110 5589 raft_consensus.cc:482] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:44:22.447127 5589 raft_consensus.cc:3037] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:22.447803 5589 raft_consensus.cc:504] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 } } peers { permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 } } peers { permanent_uuid: "1d774642ccb343338152ff1efd12d036" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34875 } }
I20240506 15:44:22.447932 5589 leader_election.cc:288] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [CANDIDATE]: Term 1 election: Requested vote from peers c1ce20ee78b041d1a88330291c2e0b42 (127.17.240.194:45861), 8cfd2c902d154db997d8f5c01be40a2d (127.17.240.195:33841)
I20240506 15:44:22.448089 5388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32613bf8208142eda18e9aed5f921ad5" candidate_uuid: "1d774642ccb343338152ff1efd12d036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ce20ee78b041d1a88330291c2e0b42"
I20240506 15:44:22.448129 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32613bf8208142eda18e9aed5f921ad5" candidate_uuid: "1d774642ccb343338152ff1efd12d036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cfd2c902d154db997d8f5c01be40a2d"
I20240506 15:44:22.448171 5388 raft_consensus.cc:3037] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:22.448189 5522 raft_consensus.cc:3037] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:22.448864 5522 raft_consensus.cc:2454] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d774642ccb343338152ff1efd12d036 in term 1.
I20240506 15:44:22.448930 5388 raft_consensus.cc:2454] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d774642ccb343338152ff1efd12d036 in term 1.
I20240506 15:44:22.449044 5188 leader_election.cc:302] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d774642ccb343338152ff1efd12d036, 8cfd2c902d154db997d8f5c01be40a2d; no voters:
I20240506 15:44:22.449152 5589 raft_consensus.cc:2781] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:22.449311 5589 raft_consensus.cc:686] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 1 LEADER]: Becoming Leader. State: Replica: 1d774642ccb343338152ff1efd12d036, State: Running, Role: LEADER
I20240506 15:44:22.449391 5589 consensus_queue.cc:237] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 } } peers { permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 } } peers { permanent_uuid: "1d774642ccb343338152ff1efd12d036" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34875 } }
I20240506 15:44:22.450117 5109 catalog_manager.cc:5484] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d774642ccb343338152ff1efd12d036 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "1d774642ccb343338152ff1efd12d036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d774642ccb343338152ff1efd12d036" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 34875 } health_report { overall_health: HEALTHY } } }
I20240506 15:44:22.477222 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.475346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.17.240.193:34875)
0506 15:44:22.475422 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.475424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.475432 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:44:22.475472 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.475472 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.475488 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.475551 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:22.475775 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.476258 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.476788 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.476899 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.476934 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:22.477023 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:22.477081 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.477144 (+ 63us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.477164 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240506 15:44:22.478843 5522 raft_consensus.cc:1261] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d [term 1 FOLLOWER]: Refusing update from remote peer 1d774642ccb343338152ff1efd12d036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:22.478843 5388 raft_consensus.cc:1261] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42 [term 1 FOLLOWER]: Refusing update from remote peer 1d774642ccb343338152ff1efd12d036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:22.479167 5589 consensus_queue.cc:1035] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cfd2c902d154db997d8f5c01be40a2d" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 33841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:22.479290 5592 consensus_queue.cc:1035] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1ce20ee78b041d1a88330291c2e0b42" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:22.482422 5596 mvcc.cc:205] Tried to move back new op lower bound from 7024682035110846464 to 7024682034993278976. Current Snapshot: MvccSnapshot[applied={T|T < 7024682035110846464 or (T in {7024682035110846464})}]
I20240506 15:44:22.482781 5597 mvcc.cc:205] Tried to move back new op lower bound from 7024682035110846464 to 7024682034993278976. Current Snapshot: MvccSnapshot[applied={T|T < 7024682035110846464 or (T in {7024682035110846464})}]
I20240506 15:44:22.483963 5109 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59096:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:22.484050 5109 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59096:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:22.484768 5109 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 33b3f69494ed4d5198a46bfc3e9b897a: Sending DeleteTablet for 3 replicas of tablet 32613bf8208142eda18e9aed5f921ad5
I20240506 15:44:22.485098 5502 tablet_service.cc:1514] Processing DeleteTablet for tablet 32613bf8208142eda18e9aed5f921ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:22 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:42882
I20240506 15:44:22.485157 5234 tablet_service.cc:1514] Processing DeleteTablet for tablet 32613bf8208142eda18e9aed5f921ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:22 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:57138
I20240506 15:44:22.485388 5605 tablet_replica.cc:330] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036: stopping tablet replica
I20240506 15:44:22.485391 5368 tablet_service.cc:1514] Processing DeleteTablet for tablet 32613bf8208142eda18e9aed5f921ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:22 UTC) from {username='kudu', principal='kudu/127.17.240.254@KRBTEST.COM'} at 127.0.0.1:56536
I20240506 15:44:22.485476 5605 raft_consensus.cc:2229] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:44:22.485563 5605 raft_consensus.cc:2258] T 32613bf8208142eda18e9aed5f921ad5 P 1d774642ccb343338152ff1efd12d036 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:22.485687 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 5171
I20240506 15:44:22.486337 5606 tablet_replica.cc:330] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: stopping tablet replica
I20240506 15:44:22.486449 5606 raft_consensus.cc:2229] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:44:22.486546 5606 raft_consensus.cc:2258] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:22.486958 5606 ts_tablet_manager.cc:1875] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:44:22.488978 5606 ts_tablet_manager.cc:1888] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:44:22.489058 5606 log.cc:1196] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715009978772892-18371-0/minicluster-data/ts-1/wal/wals/32613bf8208142eda18e9aed5f921ad5
I20240506 15:44:22.489360 5606 ts_tablet_manager.cc:1909] T 32613bf8208142eda18e9aed5f921ad5 P c1ce20ee78b041d1a88330291c2e0b42: Deleting consensus metadata
I20240506 15:44:22.489758 5095 catalog_manager.cc:4830] TS c1ce20ee78b041d1a88330291c2e0b42 (127.17.240.194:45861): tablet 32613bf8208142eda18e9aed5f921ad5 (table test-table [id=4aa4488d1d6e4f1a9ada10e849ac05ce]) successfully deleted
I20240506 15:44:22.492993 5604 tablet_replica.cc:330] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d: stopping tablet replica
W20240506 15:44:22.492971 5093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:34875 (error 108)
I20240506 15:44:22.493088 5604 raft_consensus.cc:2229] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:44:22.493136 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 5305
I20240506 15:44:22.493196 5604 raft_consensus.cc:2258] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:22.493446 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.493340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:34875 (local address 127.0.0.1:57166)
0506 15:44:22.493384 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.493412 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240506 15:44:22.493938 5093 catalog_manager.cc:4573] TS 1d774642ccb343338152ff1efd12d036 (127.17.240.193:34875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32613bf8208142eda18e9aed5f921ad5: Network error: Client connection negotiation failed: client connection to 127.17.240.193:34875: connect: Connection refused (error 111)
I20240506 15:44:22.497810 5604 ts_tablet_manager.cc:1875] T 32613bf8208142eda18e9aed5f921ad5 P 8cfd2c902d154db997d8f5c01be40a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:44:22.498628 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 5439
I20240506 15:44:22.503217 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 5077
I20240506 15:44:22.503315 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.503173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:33841 (local address 127.0.0.1:42884)
0506 15:44:22.503240 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.503274 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.195:33841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
2024-05-06T15:44:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1867 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2024-05-06T15:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:22Z Disabled control of system clock
I20240506 15:44:22.530386 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:34221
--webserver_interface=127.17.240.254
--webserver_port=0
--builtin_ntp_servers=127.17.240.212:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:34221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240506 15:44:22.607878 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:22.608065 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:22.608086 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:22.609369 5611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:22.609411 5611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:22.609422 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:22.609434 5611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:22.609447 5611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:22.610885 5611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:34221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:34221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:22.611126 5611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:22.611366 5611 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:22.616350 5617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:22.617044 5616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:22.617640 5619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:22.618165 5611 server_base.cc:1034] running on GCE node
I20240506 15:44:22.618378 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:22.618680 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:22.619819 5611 hybrid_clock.cc:648] HybridClock initialized: now 1715010262619808 us; error 27 us; skew 500 ppm
I20240506 15:44:22.621001 5611 webserver.cc:460] Webserver started at http://127.17.240.254:45029/ using document root <none> and password file <none>
I20240506 15:44:22.621207 5611 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:22.621249 5611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:22.621356 5611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:22.622399 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "9a21d23ba79f2e1551f66e3b80b4935c"
server_key_iv: "e8d69f94a70e7e56ac5a2b1ff7889d14"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.622746 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "9a21d23ba79f2e1551f66e3b80b4935c"
server_key_iv: "e8d69f94a70e7e56ac5a2b1ff7889d14"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.623981 5611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:44:22.624668 5625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.624850 5611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:22.624929 5611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "9a21d23ba79f2e1551f66e3b80b4935c"
server_key_iv: "e8d69f94a70e7e56ac5a2b1ff7889d14"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.624994 5611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:22.635784 5611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:22.636036 5611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:22.636147 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:22.639967 5611 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:34221
I20240506 15:44:22.639995 5677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:34221 every 8 connection(s)
I20240506 15:44:22.640300 5611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:22.640887 5678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:22.643501 5678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea: Bootstrap starting.
I20240506 15:44:22.644063 5678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:22.644243 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 5611
I20240506 15:44:22.644345 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:22.644338 5678 log.cc:824] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:22.644562 18371 external_mini_cluster.cc:1428] Setting key b00bf8118db5043f7bdc4411aa9eb976
I20240506 15:44:22.645071 5678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea: No bootstrap required, opened a new log
I20240506 15:44:22.647073 5678 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 34221 } }
I20240506 15:44:22.647213 5678 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:22.647231 5678 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f9bd07d71040af9a6b48d8a8bb94ea, State: Initialized, Role: FOLLOWER
I20240506 15:44:22.647375 5678 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 34221 } }
I20240506 15:44:22.647454 5678 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:22.647490 5678 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:22.647541 5678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:22.648315 5678 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 34221 } }
I20240506 15:44:22.648413 5678 leader_election.cc:302] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8f9bd07d71040af9a6b48d8a8bb94ea; no voters:
I20240506 15:44:22.648582 5678 leader_election.cc:288] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:22.648808 5683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:22.648854 5678 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:22.648989 5683 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [term 1 LEADER]: Becoming Leader. State: Replica: b8f9bd07d71040af9a6b48d8a8bb94ea, State: Running, Role: LEADER
I20240506 15:44:22.649083 5683 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 34221 } }
I20240506 15:44:22.649422 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f9bd07d71040af9a6b48d8a8bb94ea. Latest consensus state: current_term: 1 leader_uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 34221 } } }
I20240506 15:44:22.649499 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:22.649654 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9bd07d71040af9a6b48d8a8bb94ea" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 34221 } } }
I20240506 15:44:22.649724 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:22.649709 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.645077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.17.240.254:34221)
0506 15:44:22.645272 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.645278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.645297 (+ 19us) server_negotiation.cc:407] Connection header received
0506 15:44:22.645756 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.645762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.645813 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.645937 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:22.646403 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.647198 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.648024 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.648203 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.648752 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:22.648764 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:22.648764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:22.649015 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:22.649123 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:22.649132 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:22.649406 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.649455 (+ 49us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.649512 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20240506 15:44:22.649880 5692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:22.650344 5692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:22.652024 5692 catalog_manager.cc:1347] Generated new cluster ID: 09c4132c436d48e6ba9fb81b572c2009
I20240506 15:44:22.652078 5692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:22.655143 5692 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:22.655652 5692 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:22.667023 5692 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea: Generated new TSK 0
I20240506 15:44:22.667222 5692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:22.676297 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.193:0
--local_ip_for_outbound_sockets=127.17.240.193
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:34221
--builtin_ntp_servers=127.17.240.212:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240506 15:44:22.757404 5702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:22.757591 5702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:22.757614 5702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:22.758962 5702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:22.759044 5702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:22.759094 5702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.193
I20240506 15:44:22.760668 5702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.240.193
--webserver_port=0
--tserver_master_addrs=127.17.240.254:34221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.193
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:22.760886 5702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:22.761164 5702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:22.766657 5707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:22.767171 5708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:22.767621 5710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:22.768543 5702 server_base.cc:1034] running on GCE node
I20240506 15:44:22.768847 5702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:22.769091 5702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:22.770242 5702 hybrid_clock.cc:648] HybridClock initialized: now 1715010262770228 us; error 35 us; skew 500 ppm
I20240506 15:44:22.771780 5702 webserver.cc:460] Webserver started at http://127.17.240.193:33827/ using document root <none> and password file <none>
I20240506 15:44:22.771996 5702 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:22.772037 5702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:22.772140 5702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:22.773169 5702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/data/instance:
uuid: "0a71f23c80554054858aa689e5265c01"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "8d1aec20cf9d311841922d7b16da179b"
server_key_iv: "6a42251907918ef0ef8bf919ded9eedc"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.773463 5702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance:
uuid: "0a71f23c80554054858aa689e5265c01"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "8d1aec20cf9d311841922d7b16da179b"
server_key_iv: "6a42251907918ef0ef8bf919ded9eedc"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.774649 5702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:44:22.775390 5716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.775550 5702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.775631 5702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/wal
uuid: "0a71f23c80554054858aa689e5265c01"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "8d1aec20cf9d311841922d7b16da179b"
server_key_iv: "6a42251907918ef0ef8bf919ded9eedc"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.775702 5702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:22.789450 5702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:22.789700 5702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:22.789808 5702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:22.790014 5702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:22.790369 5702 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:22.790405 5702 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.790428 5702 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:22.790441 5702 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.797274 5702 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.193:46671
I20240506 15:44:22.797319 5829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.193:46671 every 8 connection(s)
I20240506 15:44:22.797623 5702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/data/info.pb
I20240506 15:44:22.801004 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 5702
I20240506 15:44:22.801115 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-0/wal/instance
I20240506 15:44:22.801287 18371 external_mini_cluster.cc:1428] Setting key a730c60ae5b71b326bb807513cf03db1
I20240506 15:44:22.802124 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.798265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:42325 (local address 127.17.240.254:34221)
0506 15:44:22.798348 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.798351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.798533 (+ 182us) server_negotiation.cc:407] Connection header received
0506 15:44:22.798992 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.798994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.799027 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.799097 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:22.799548 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.800030 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.800545 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.800649 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.801831 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:22.801834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:22.801834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:22.801838 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:22.801853 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:22.801857 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:22.801915 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.802008 (+ 93us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.802047 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20240506 15:44:22.802235 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.798234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:34221 (local address 127.17.240.193:42325)
0506 15:44:22.798419 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.798443 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:22.798906 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:22.799149 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:22.799153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:22.799161 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:22.799477 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.799481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.800096 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.800096 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:22.800458 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.800459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.800584 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.801697 (+ 1113us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:44:22.801698 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:22.801725 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:44:22.801746 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:22.801925 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:22.801928 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:22.801931 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:44:22.802009 (+ 78us) client_negotiation.cc:240] Negotiation successful
0506 15:44:22.802087 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20240506 15:44:22.802354 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.194:0
--local_ip_for_outbound_sockets=127.17.240.194
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:34221
--builtin_ntp_servers=127.17.240.212:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240506 15:44:22.802803 5830 heartbeater.cc:347] Connected to a master server at 127.17.240.254:34221
I20240506 15:44:22.802903 5830 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:22.803149 5830 heartbeater.cc:511] Master 127.17.240.254:34221 requested a full tablet report, sending...
I20240506 15:44:22.803632 5642 ts_manager.cc:194] Registered new tserver with Master: 0a71f23c80554054858aa689e5265c01 (127.17.240.193:46671)
I20240506 15:44:22.804260 5642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.193:42325
W20240506 15:44:22.881935 5833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:22.882107 5833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:22.882129 5833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:22.883437 5833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:22.883484 5833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:22.883508 5833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.194
I20240506 15:44:22.884882 5833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.240.194
--webserver_port=0
--tserver_master_addrs=127.17.240.254:34221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.194
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:22.885094 5833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:22.885326 5833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:22.890151 5838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:22.890676 5839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:22.891237 5841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:22.891919 5833 server_base.cc:1034] running on GCE node
I20240506 15:44:22.892103 5833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:22.892302 5833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:22.893433 5833 hybrid_clock.cc:648] HybridClock initialized: now 1715010262893417 us; error 31 us; skew 500 ppm
I20240506 15:44:22.894646 5833 webserver.cc:460] Webserver started at http://127.17.240.194:41227/ using document root <none> and password file <none>
I20240506 15:44:22.894842 5833 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:22.894887 5833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:22.895044 5833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:22.896024 5833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/data/instance:
uuid: "26cf1d49b5a44e339fde40dc4742b989"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "52a6273d6bad2dd5599bf189693079f7"
server_key_iv: "3ad8590904ea1ec49f125b96957cad8f"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.896327 5833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance:
uuid: "26cf1d49b5a44e339fde40dc4742b989"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "52a6273d6bad2dd5599bf189693079f7"
server_key_iv: "3ad8590904ea1ec49f125b96957cad8f"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.897472 5833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:44:22.898142 5847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.898326 5833 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:22.898404 5833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/wal
uuid: "26cf1d49b5a44e339fde40dc4742b989"
format_stamp: "Formatted at 2024-05-06 15:44:22 on dist-test-slave-v0k1"
server_key: "52a6273d6bad2dd5599bf189693079f7"
server_key_iv: "3ad8590904ea1ec49f125b96957cad8f"
server_key_version: "encryptionkey@0"
I20240506 15:44:22.898487 5833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:22.909085 5833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:22.909304 5833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:22.909409 5833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:22.909588 5833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:22.909876 5833 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:22.909911 5833 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.909947 5833 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:22.909982 5833 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:22.915128 5833 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.194:40039
I20240506 15:44:22.915166 5960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.194:40039 every 8 connection(s)
I20240506 15:44:22.915443 5833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/data/info.pb
I20240506 15:44:22.917110 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 5833
I20240506 15:44:22.917205 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-1/wal/instance
I20240506 15:44:22.917373 18371 external_mini_cluster.cc:1428] Setting key 788c0d17418707ff73b1dba3431a53dd
I20240506 15:44:22.918278 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.240.195:0
--local_ip_for_outbound_sockets=127.17.240.195
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:34221
--builtin_ntp_servers=127.17.240.212:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240506 15:44:22.919889 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.916116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.194:41601 (local address 127.17.240.254:34221)
0506 15:44:22.916180 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:22.916182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:22.916354 (+ 172us) server_negotiation.cc:407] Connection header received
0506 15:44:22.916801 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:22.916803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:22.916812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:22.916849 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:22.917343 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.917977 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.918538 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.918747 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.919609 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:22.919613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:22.919614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:22.919619 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:22.919636 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:22.919642 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:22.919708 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:22.919804 (+ 96us) server_negotiation.cc:299] Negotiation successful
0506 15:44:22.919831 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:44:22.920166 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:22.916076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:34221 (local address 127.17.240.194:41601)
0506 15:44:22.916258 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:22.916285 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:22.916706 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:22.916900 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:22.916904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:22.916911 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:22.917236 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.917243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.918049 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:22.918050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:22.918450 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:22.918452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:22.918572 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:22.919468 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:44:22.919468 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:22.919487 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:44:22.919504 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:22.919721 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:22.919725 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:22.919727 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:44:22.919805 (+ 78us) client_negotiation.cc:240] Negotiation successful
0506 15:44:22.919886 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20240506 15:44:22.920625 5961 heartbeater.cc:347] Connected to a master server at 127.17.240.254:34221
I20240506 15:44:22.920715 5961 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:22.920904 5961 heartbeater.cc:511] Master 127.17.240.254:34221 requested a full tablet report, sending...
I20240506 15:44:22.921296 5642 ts_manager.cc:194] Registered new tserver with Master: 26cf1d49b5a44e339fde40dc4742b989 (127.17.240.194:40039)
I20240506 15:44:22.921680 5642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.194:41601
W20240506 15:44:22.997784 5964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:22.997982 5964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:22.998018 5964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:22.999404 5964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:22.999452 5964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:22.999497 5964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.240.195
I20240506 15:44:23.000965 5964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.240.195
--webserver_port=0
--tserver_master_addrs=127.17.240.254:34221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.240.195
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:23.001220 5964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:23.001497 5964 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:23.006556 5972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:23.007000 5969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:23.007699 5970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:23.008404 5964 server_base.cc:1034] running on GCE node
I20240506 15:44:23.008580 5964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:23.008797 5964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:23.009940 5964 hybrid_clock.cc:648] HybridClock initialized: now 1715010263009931 us; error 27 us; skew 500 ppm
I20240506 15:44:23.011255 5964 webserver.cc:460] Webserver started at http://127.17.240.195:45697/ using document root <none> and password file <none>
I20240506 15:44:23.011461 5964 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:23.011518 5964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:23.011632 5964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:23.012630 5964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/data/instance:
uuid: "37065deb44a646c9b1b7526c852301c4"
format_stamp: "Formatted at 2024-05-06 15:44:23 on dist-test-slave-v0k1"
server_key: "a10ad4a757522dc422b3cf934f8889fd"
server_key_iv: "11e6025577a4b79a7b06ebd79808fb0c"
server_key_version: "encryptionkey@0"
I20240506 15:44:23.012954 5964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance:
uuid: "37065deb44a646c9b1b7526c852301c4"
format_stamp: "Formatted at 2024-05-06 15:44:23 on dist-test-slave-v0k1"
server_key: "a10ad4a757522dc422b3cf934f8889fd"
server_key_iv: "11e6025577a4b79a7b06ebd79808fb0c"
server_key_version: "encryptionkey@0"
I20240506 15:44:23.014139 5964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:44:23.014798 5978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:23.014954 5964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:23.015053 5964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/wal
uuid: "37065deb44a646c9b1b7526c852301c4"
format_stamp: "Formatted at 2024-05-06 15:44:23 on dist-test-slave-v0k1"
server_key: "a10ad4a757522dc422b3cf934f8889fd"
server_key_iv: "11e6025577a4b79a7b06ebd79808fb0c"
server_key_version: "encryptionkey@0"
I20240506 15:44:23.015117 5964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:23.026917 5964 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:23.027163 5964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:23.027276 5964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:23.027454 5964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:44:23.027707 5964 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:44:23.027736 5964 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:23.027757 5964 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:44:23.027776 5964 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:23.032943 5964 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.195:35167
I20240506 15:44:23.033003 6091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.195:35167 every 8 connection(s)
I20240506 15:44:23.033288 5964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/data/info.pb
I20240506 15:44:23.037328 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.033965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.195:34437 (local address 127.17.240.254:34221)
0506 15:44:23.034016 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:23.034018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:23.034204 (+ 186us) server_negotiation.cc:407] Connection header received
0506 15:44:23.034644 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:23.034645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:23.034652 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:23.034712 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:23.035173 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.035648 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.036122 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.036222 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.037101 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:23.037106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:23.037106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:23.037110 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:23.037124 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:23.037127 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:23.037172 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:23.037242 (+ 70us) server_negotiation.cc:299] Negotiation successful
0506 15:44:23.037269 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240506 15:44:23.037432 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.033933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:34221 (local address 127.17.240.195:34437)
0506 15:44:23.034109 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:23.034137 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:23.034565 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:23.034766 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:23.034769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:23.034777 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:23.035109 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.035113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.035710 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.035710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:23.036066 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.036066 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.036157 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.036986 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0506 15:44:23.036987 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:23.037003 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0506 15:44:23.037019 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:23.037179 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:23.037181 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:23.037182 (+ 1us) client_negotiation.cc:769] Sending connection context
0506 15:44:23.037236 (+ 54us) client_negotiation.cc:240] Negotiation successful
0506 15:44:23.037290 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20240506 15:44:23.037901 6092 heartbeater.cc:347] Connected to a master server at 127.17.240.254:34221
I20240506 15:44:23.037989 6092 heartbeater.cc:464] Registering TS with master...
I20240506 15:44:23.038184 6092 heartbeater.cc:511] Master 127.17.240.254:34221 requested a full tablet report, sending...
I20240506 15:44:23.038537 5642 ts_manager.cc:194] Registered new tserver with Master: 37065deb44a646c9b1b7526c852301c4 (127.17.240.195:35167)
I20240506 15:44:23.038870 5642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.240.195:34437
I20240506 15:44:23.043107 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 5964
I20240506 15:44:23.043186 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/wal/instance
I20240506 15:44:23.043327 18371 external_mini_cluster.cc:1428] Setting key 8b20fe8d7d7807ee0899e5b965a2a3d7
I20240506 15:44:23.044238 18371 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 15:44:23.047756 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.045489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.17.240.254:34221)
0506 15:44:23.045536 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:23.045538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:23.045634 (+ 96us) server_negotiation.cc:407] Connection header received
0506 15:44:23.045688 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:23.045689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:23.045697 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:23.045755 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:23.046026 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.046485 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.046979 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.047100 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.047559 (+ 459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:23.047562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:23.047563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:23.047567 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:23.047580 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:23.047584 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:23.047627 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:23.047665 (+ 38us) server_negotiation.cc:299] Negotiation successful
0506 15:44:23.047690 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:44:23.049336 5642 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:44:23.050855 5642 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:44:23.056093 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.052727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:46671 (local address 127.0.0.1:46240)
0506 15:44:23.052998 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:23.053026 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:23.053097 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:23.053404 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:23.053406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:23.053409 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:23.053697 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.053698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.055074 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.055074 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:23.055892 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.055894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.055951 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.055953 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:44:23.055983 (+ 30us) client_negotiation.cc:240] Negotiation successful
0506 15:44:23.056015 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":64,"threads_started":1}
I20240506 15:44:23.056603 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.052869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.17.240.194:40039)
0506 15:44:23.053040 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:23.053050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:23.053154 (+ 104us) server_negotiation.cc:407] Connection header received
0506 15:44:23.053288 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:23.053290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:23.053305 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:23.053392 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:23.053955 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.055102 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.055958 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.056435 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.056498 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:23.056526 (+ 28us) server_negotiation.cc:299] Negotiation successful
0506 15:44:23.056549 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20240506 15:44:23.056799 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.053017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.17.240.195:35167)
0506 15:44:23.053185 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:23.053193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:23.053572 (+ 379us) server_negotiation.cc:407] Connection header received
0506 15:44:23.053712 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:23.053714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:23.053734 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:23.053816 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:23.054097 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.054801 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.056069 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.056586 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.056660 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:23.056694 (+ 34us) server_negotiation.cc:299] Negotiation successful
0506 15:44:23.056727 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20240506 15:44:23.056955 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.052831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.17.240.193:46671)
0506 15:44:23.053223 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:23.053230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:23.053243 (+ 13us) server_negotiation.cc:407] Connection header received
0506 15:44:23.053285 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:23.053287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:23.053305 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:23.053392 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:23.053770 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.054956 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.056243 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.056747 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.056811 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:23.056841 (+ 30us) server_negotiation.cc:299] Negotiation successful
0506 15:44:23.056884 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":93,"threads_started":1}
I20240506 15:44:23.056123 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.052767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:40039 (local address 127.0.0.1:49480)
0506 15:44:23.053090 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:23.053097 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:23.053183 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:23.053403 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:23.053406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:23.053410 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:23.053697 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.053698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.055180 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.055181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:23.055883 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.055885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.055948 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.055951 (+ 3us) client_negotiation.cc:769] Sending connection context
0506 15:44:23.055982 (+ 31us) client_negotiation.cc:240] Negotiation successful
0506 15:44:23.056025 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":90,"threads_started":1}
I20240506 15:44:23.056944 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.052944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:35167 (local address 127.0.0.1:44890)
0506 15:44:23.053497 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:23.053505 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:23.053621 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:23.053809 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:23.053810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:23.053812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:23.054023 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.054024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.054878 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.054879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:23.055769 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.055771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.055835 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.055875 (+ 40us) client_negotiation.cc:769] Sending connection context
0506 15:44:23.055941 (+ 66us) client_negotiation.cc:240] Negotiation successful
0506 15:44:23.055986 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":68,"threads_started":1}
I20240506 15:44:23.057677 5895 tablet_service.cc:1467] Processing CreateTablet for tablet cff593880f794d28a929561f32ae210f (DEFAULT_TABLE table=test-table [id=c6f54c0e15f24989b21054cdd5396cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:23.057663 5764 tablet_service.cc:1467] Processing CreateTablet for tablet cff593880f794d28a929561f32ae210f (DEFAULT_TABLE table=test-table [id=c6f54c0e15f24989b21054cdd5396cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:23.057736 6026 tablet_service.cc:1467] Processing CreateTablet for tablet cff593880f794d28a929561f32ae210f (DEFAULT_TABLE table=test-table [id=c6f54c0e15f24989b21054cdd5396cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 15:44:23.057988 6026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff593880f794d28a929561f32ae210f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:23.057993 5764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff593880f794d28a929561f32ae210f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:23.057992 5895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff593880f794d28a929561f32ae210f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:23.060364 6106 tablet_bootstrap.cc:492] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: Bootstrap starting.
I20240506 15:44:23.060958 6106 tablet_bootstrap.cc:654] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:23.061152 6108 tablet_bootstrap.cc:492] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01: Bootstrap starting.
I20240506 15:44:23.061373 6106 log.cc:824] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:23.061724 6108 tablet_bootstrap.cc:654] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:23.062006 6108 log.cc:824] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:23.062125 6106 tablet_bootstrap.cc:492] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: No bootstrap required, opened a new log
I20240506 15:44:23.062197 6106 ts_tablet_manager.cc:1367] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240506 15:44:23.062237 6107 tablet_bootstrap.cc:492] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989: Bootstrap starting.
I20240506 15:44:23.062793 6107 tablet_bootstrap.cc:654] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:23.063117 6107 log.cc:824] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:23.063513 6108 tablet_bootstrap.cc:492] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01: No bootstrap required, opened a new log
I20240506 15:44:23.063589 6108 ts_tablet_manager.cc:1367] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 15:44:23.063854 6107 tablet_bootstrap.cc:492] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989: No bootstrap required, opened a new log
I20240506 15:44:23.063911 6107 ts_tablet_manager.cc:1367] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240506 15:44:23.064443 6106 raft_consensus.cc:348] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a71f23c80554054858aa689e5265c01" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46671 } } peers { permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 } } peers { permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 } }
I20240506 15:44:23.064555 6106 raft_consensus.cc:374] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:23.064582 6106 raft_consensus.cc:729] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37065deb44a646c9b1b7526c852301c4, State: Initialized, Role: FOLLOWER
I20240506 15:44:23.064677 6106 consensus_queue.cc:260] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a71f23c80554054858aa689e5265c01" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46671 } } peers { permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 } } peers { permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 } }
I20240506 15:44:23.064919 6106 ts_tablet_manager.cc:1398] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:23.064994 6092 heartbeater.cc:503] Master 127.17.240.254:34221 was elected leader, sending a full tablet report...
I20240506 15:44:23.066213 6107 raft_consensus.cc:348] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a71f23c80554054858aa689e5265c01" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46671 } } peers { permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 } } peers { permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 } }
I20240506 15:44:23.066221 6108 raft_consensus.cc:348] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a71f23c80554054858aa689e5265c01" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46671 } } peers { permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 } } peers { permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 } }
I20240506 15:44:23.066354 6108 raft_consensus.cc:374] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:23.066354 6107 raft_consensus.cc:374] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:23.066381 6107 raft_consensus.cc:729] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cf1d49b5a44e339fde40dc4742b989, State: Initialized, Role: FOLLOWER
I20240506 15:44:23.066381 6108 raft_consensus.cc:729] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a71f23c80554054858aa689e5265c01, State: Initialized, Role: FOLLOWER
I20240506 15:44:23.066483 6108 consensus_queue.cc:260] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a71f23c80554054858aa689e5265c01" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46671 } } peers { permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 } } peers { permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 } }
I20240506 15:44:23.066504 6107 consensus_queue.cc:260] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a71f23c80554054858aa689e5265c01" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46671 } } peers { permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 } } peers { permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 } }
I20240506 15:44:23.066733 6107 ts_tablet_manager.cc:1398] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:23.066733 6108 ts_tablet_manager.cc:1398] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 15:44:23.066782 5830 heartbeater.cc:503] Master 127.17.240.254:34221 was elected leader, sending a full tablet report...
I20240506 15:44:23.066846 5961 heartbeater.cc:503] Master 127.17.240.254:34221 was elected leader, sending a full tablet report...
I20240506 15:44:23.083987 6114 raft_consensus.cc:482] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:44:23.084117 6114 raft_consensus.cc:504] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a71f23c80554054858aa689e5265c01" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46671 } } peers { permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 } } peers { permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 } }
I20240506 15:44:23.084477 6114 leader_election.cc:288] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37065deb44a646c9b1b7526c852301c4 (127.17.240.195:35167), 26cf1d49b5a44e339fde40dc4742b989 (127.17.240.194:40039)
I20240506 15:44:23.087361 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.084597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.195:35167 (local address 127.17.240.193:53811)
0506 15:44:23.084639 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:23.084651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:23.084697 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:23.084900 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:23.084902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:23.084904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:23.085128 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.085131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.086073 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.086074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:23.087046 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.087050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.087127 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.087131 (+ 4us) client_negotiation.cc:769] Sending connection context
0506 15:44:23.087167 (+ 36us) client_negotiation.cc:240] Negotiation successful
0506 15:44:23.087209 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20240506 15:44:23.087500 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.084630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:40039 (local address 127.17.240.193:53341)
0506 15:44:23.084838 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:23.084846 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:23.084894 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:23.085021 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:23.085022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:23.085023 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0506 15:44:23.085248 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.085249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.086105 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.086105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:23.087059 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:23.087061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.087142 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.087144 (+ 2us) client_negotiation.cc:769] Sending connection context
0506 15:44:23.087178 (+ 34us) client_negotiation.cc:240] Negotiation successful
0506 15:44:23.087206 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
I20240506 15:44:23.087723 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.084691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:53811 (local address 127.17.240.195:35167)
0506 15:44:23.084757 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:23.084759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:23.084769 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:44:23.084799 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:23.084800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:23.084809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:23.084885 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:23.085201 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.085973 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.087134 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.087591 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.087602 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:23.087627 (+ 25us) server_negotiation.cc:299] Negotiation successful
0506 15:44:23.087666 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240506 15:44:23.087935 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.084792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.240.193:53341 (local address 127.17.240.194:40039)
0506 15:44:23.084873 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:23.084875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:23.084884 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:44:23.084959 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:23.084960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:23.084967 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:23.085008 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0506 15:44:23.085308 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.086039 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.087325 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.087802 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.087810 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:23.087834 (+ 24us) server_negotiation.cc:299] Negotiation successful
0506 15:44:23.087866 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240506 15:44:23.088035 6046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff593880f794d28a929561f32ae210f" candidate_uuid: "0a71f23c80554054858aa689e5265c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37065deb44a646c9b1b7526c852301c4" is_pre_election: true
I20240506 15:44:23.088205 6046 raft_consensus.cc:2454] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a71f23c80554054858aa689e5265c01 in term 0.
I20240506 15:44:23.088236 5915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff593880f794d28a929561f32ae210f" candidate_uuid: "0a71f23c80554054858aa689e5265c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cf1d49b5a44e339fde40dc4742b989" is_pre_election: true
I20240506 15:44:23.088383 5915 raft_consensus.cc:2454] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a71f23c80554054858aa689e5265c01 in term 0.
I20240506 15:44:23.088625 5717 leader_election.cc:302] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a71f23c80554054858aa689e5265c01, 37065deb44a646c9b1b7526c852301c4; no voters:
I20240506 15:44:23.088754 6114 raft_consensus.cc:2781] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:44:23.088822 6114 raft_consensus.cc:482] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:44:23.088840 6114 raft_consensus.cc:3037] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:23.089524 6114 raft_consensus.cc:504] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a71f23c80554054858aa689e5265c01" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46671 } } peers { permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 } } peers { permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 } }
I20240506 15:44:23.089666 6114 leader_election.cc:288] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [CANDIDATE]: Term 1 election: Requested vote from peers 37065deb44a646c9b1b7526c852301c4 (127.17.240.195:35167), 26cf1d49b5a44e339fde40dc4742b989 (127.17.240.194:40039)
I20240506 15:44:23.089819 5915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff593880f794d28a929561f32ae210f" candidate_uuid: "0a71f23c80554054858aa689e5265c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cf1d49b5a44e339fde40dc4742b989"
I20240506 15:44:23.089885 5915 raft_consensus.cc:3037] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:23.090557 5915 raft_consensus.cc:2454] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a71f23c80554054858aa689e5265c01 in term 1.
I20240506 15:44:23.090669 6046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff593880f794d28a929561f32ae210f" candidate_uuid: "0a71f23c80554054858aa689e5265c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37065deb44a646c9b1b7526c852301c4"
I20240506 15:44:23.090726 5720 leader_election.cc:302] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a71f23c80554054858aa689e5265c01, 26cf1d49b5a44e339fde40dc4742b989; no voters:
I20240506 15:44:23.090752 6046 raft_consensus.cc:3037] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:23.090808 6114 raft_consensus.cc:2781] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:23.090955 6114 raft_consensus.cc:686] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [term 1 LEADER]: Becoming Leader. State: Replica: 0a71f23c80554054858aa689e5265c01, State: Running, Role: LEADER
I20240506 15:44:23.091089 6114 consensus_queue.cc:237] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a71f23c80554054858aa689e5265c01" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46671 } } peers { permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 } } peers { permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 } }
I20240506 15:44:23.091722 6046 raft_consensus.cc:2454] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a71f23c80554054858aa689e5265c01 in term 1.
I20240506 15:44:23.091936 5642 catalog_manager.cc:5484] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a71f23c80554054858aa689e5265c01 (127.17.240.193). New cstate: current_term: 1 leader_uuid: "0a71f23c80554054858aa689e5265c01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a71f23c80554054858aa689e5265c01" member_type: VOTER last_known_addr { host: "127.17.240.193" port: 46671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 } health_report { overall_health: UNKNOWN } } }
I20240506 15:44:23.105240 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.103389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.17.240.193:46671)
0506 15:44:23.103447 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:23.103448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:23.103455 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:23.103492 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:23.103493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:23.103509 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:23.103576 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:23.103800 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.104290 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:23.104818 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:23.104935 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:23.104969 (+ 34us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:23.105055 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:23.105114 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:23.105162 (+ 48us) server_negotiation.cc:299] Negotiation successful
0506 15:44:23.105181 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240506 15:44:23.106972 5915 raft_consensus.cc:1261] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989 [term 1 FOLLOWER]: Refusing update from remote peer 0a71f23c80554054858aa689e5265c01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:23.106981 6046 raft_consensus.cc:1261] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4 [term 1 FOLLOWER]: Refusing update from remote peer 0a71f23c80554054858aa689e5265c01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:23.107349 6114 consensus_queue.cc:1035] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37065deb44a646c9b1b7526c852301c4" member_type: VOTER last_known_addr { host: "127.17.240.195" port: 35167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:23.107486 6116 consensus_queue.cc:1035] T cff593880f794d28a929561f32ae210f P 0a71f23c80554054858aa689e5265c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26cf1d49b5a44e339fde40dc4742b989" member_type: VOTER last_known_addr { host: "127.17.240.194" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:23.109740 6119 mvcc.cc:205] Tried to move back new op lower bound from 7024682037683462144 to 7024682037621739520. Current Snapshot: MvccSnapshot[applied={T|T < 7024682037683462144}]
I20240506 15:44:23.110567 6120 mvcc.cc:205] Tried to move back new op lower bound from 7024682037683462144 to 7024682037621739520. Current Snapshot: MvccSnapshot[applied={T|T < 7024682037683462144}]
I20240506 15:44:23.110615 6121 mvcc.cc:205] Tried to move back new op lower bound from 7024682037683462144 to 7024682037621739520. Current Snapshot: MvccSnapshot[applied={T|T < 7024682037683462144}]
I20240506 15:44:23.111901 5641 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52782:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:23.112013 5641 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52782:
table { table_name: "test-table" } modify_external_catalogs: true
I20240506 15:44:23.112726 5641 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b8f9bd07d71040af9a6b48d8a8bb94ea: Sending DeleteTablet for 3 replicas of tablet cff593880f794d28a929561f32ae210f
I20240506 15:44:23.113062 5895 tablet_service.cc:1514] Processing DeleteTablet for tablet cff593880f794d28a929561f32ae210f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:23 UTC) from {username='slave'} at 127.0.0.1:49480
I20240506 15:44:23.113121 6026 tablet_service.cc:1514] Processing DeleteTablet for tablet cff593880f794d28a929561f32ae210f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:23 UTC) from {username='slave'} at 127.0.0.1:44890
I20240506 15:44:23.113312 5764 tablet_service.cc:1514] Processing DeleteTablet for tablet cff593880f794d28a929561f32ae210f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 15:44:23 UTC) from {username='slave'} at 127.0.0.1:46240
I20240506 15:44:23.113519 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 5702
I20240506 15:44:23.118147 6128 tablet_replica.cc:330] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989: stopping tablet replica
I20240506 15:44:23.118263 6128 raft_consensus.cc:2229] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:44:23.118381 6128 raft_consensus.cc:2258] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:23.118558 6129 tablet_replica.cc:330] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: stopping tablet replica
I20240506 15:44:23.118659 6129 raft_consensus.cc:2229] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:44:23.118755 6129 raft_consensus.cc:2258] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:44:23.119024 6128 ts_tablet_manager.cc:1875] T cff593880f794d28a929561f32ae210f P 26cf1d49b5a44e339fde40dc4742b989: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 15:44:23.119127 5626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.240.193:46671 (error 108)
I20240506 15:44:23.119212 6129 ts_tablet_manager.cc:1875] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 15:44:23.119694 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 5833
I20240506 15:44:23.123032 6129 ts_tablet_manager.cc:1888] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 15:44:23.123112 6129 log.cc:1196] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: Deleting WAL directory at /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715009978772892-18371-0/minicluster-data/ts-2/wal/wals/cff593880f794d28a929561f32ae210f
I20240506 15:44:23.123394 6129 ts_tablet_manager.cc:1909] T cff593880f794d28a929561f32ae210f P 37065deb44a646c9b1b7526c852301c4: Deleting consensus metadata
I20240506 15:44:23.123454 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.123291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.193:46671 (local address 127.0.0.1:46250)
0506 15:44:23.123362 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:23.123407 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.193:46671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20240506 15:44:23.123575 5626 catalog_manager.cc:4573] TS 0a71f23c80554054858aa689e5265c01 (127.17.240.193:46671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cff593880f794d28a929561f32ae210f: Network error: Client connection negotiation failed: client connection to 127.17.240.193:46671: connect: Connection refused (error 111)
I20240506 15:44:23.123776 5626 catalog_manager.cc:4830] TS 37065deb44a646c9b1b7526c852301c4 (127.17.240.195:35167): tablet cff593880f794d28a929561f32ae210f (table test-table [id=c6f54c0e15f24989b21054cdd5396cea]) successfully deleted
I20240506 15:44:23.125144 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 5964
I20240506 15:44:23.125197 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:23.125083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.194:40039 (local address 127.0.0.1:49492)
0506 15:44:23.125144 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:23.125171 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.194:40039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20240506 15:44:23.129912 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 5611
2024-05-06T15:44:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (626 ms)
[----------] 6 tests from EncryptionPolicyTest (7919 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:23 dist-test-slave-v0k1 krb5kdc[6134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:23 dist-test-slave-v0k1 krb5kdc[6134](info): set up 2 sockets
May 06 15:44:23 dist-test-slave-v0k1 krb5kdc[6134](info): commencing operation
krb5kdc: starting...
W20240506 15:44:24.091055 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:24 dist-test-slave-v0k1 krb5kdc[6134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:24.209842 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:24.289634 6147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:24.289851 6147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:24.289875 6147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:24.291208 6147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:24.291249 6147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:24.291262 6147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:24.291275 6147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:24.291286 6147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:24.292908 6147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34105
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:24.293097 6147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:24.293251 6147 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:24.297953 6155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:24.298488 6152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:24.299219 6153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:24.299990 6147 server_base.cc:1034] running on GCE node
I20240506 15:44:24.300242 6147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:24.300539 6147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:24.301687 6147 hybrid_clock.cc:648] HybridClock initialized: now 1715010264301673 us; error 35 us; skew 500 ppm
May 06 15:44:24 dist-test-slave-v0k1 krb5kdc[6134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:24.304147 6147 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:24.304562 6147 webserver.cc:460] Webserver started at http://127.0.0.1:45703/ using document root <none> and password file <none>
I20240506 15:44:24.304731 6147 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:24.304767 6147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:24.304852 6147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:24.305822 6147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "d41024d08c864a6da75d14e68e4d4ec7"
format_stamp: "Formatted at 2024-05-06 15:44:24 on dist-test-slave-v0k1"
server_key: "7658718e6d11f044cf28ed93f5d7fcd2"
server_key_iv: "bddfd1bab5d9b28dac0c3e9304bd67d1"
server_key_version: "encryptionkey@0"
I20240506 15:44:24.306097 6147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "d41024d08c864a6da75d14e68e4d4ec7"
format_stamp: "Formatted at 2024-05-06 15:44:24 on dist-test-slave-v0k1"
server_key: "7658718e6d11f044cf28ed93f5d7fcd2"
server_key_iv: "bddfd1bab5d9b28dac0c3e9304bd67d1"
server_key_version: "encryptionkey@0"
I20240506 15:44:24.307269 6147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:44:24.307965 6162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:24.308153 6147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:24.308231 6147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "d41024d08c864a6da75d14e68e4d4ec7"
format_stamp: "Formatted at 2024-05-06 15:44:24 on dist-test-slave-v0k1"
server_key: "7658718e6d11f044cf28ed93f5d7fcd2"
server_key_iv: "bddfd1bab5d9b28dac0c3e9304bd67d1"
server_key_version: "encryptionkey@0"
I20240506 15:44:24.308286 6147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:24.317386 6147 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:24.319087 6147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:24.319185 6147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:24.322630 6147 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34105
I20240506 15:44:24.322677 6214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34105 every 8 connection(s)
I20240506 15:44:24.322957 6147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:24.323827 6215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:24.324126 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 6147
I20240506 15:44:24.324239 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:24.324430 18371 external_mini_cluster.cc:1428] Setting key 5c725ba4473bda6ee502c7b9dffdd6f8
I20240506 15:44:24.326239 6215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7: Bootstrap starting.
I20240506 15:44:24.326774 6215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:24.327059 6215 log.cc:824] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:24.327736 6215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7: No bootstrap required, opened a new log
May 06 15:44:24 dist-test-slave-v0k1 krb5kdc[6134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240506 15:44:24.329842 6215 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41024d08c864a6da75d14e68e4d4ec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } }
I20240506 15:44:24.329975 6215 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:24.330005 6215 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41024d08c864a6da75d14e68e4d4ec7, State: Initialized, Role: FOLLOWER
I20240506 15:44:24.330119 6215 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41024d08c864a6da75d14e68e4d4ec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } }
I20240506 15:44:24.330199 6215 raft_consensus.cc:388] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:24.330242 6215 raft_consensus.cc:482] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:24.330289 6215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:24.330953 6215 raft_consensus.cc:504] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41024d08c864a6da75d14e68e4d4ec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } }
I20240506 15:44:24.331087 6215 leader_election.cc:302] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41024d08c864a6da75d14e68e4d4ec7; no voters:
I20240506 15:44:24.331251 6215 leader_election.cc:288] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:24.331316 6220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:24.331439 6220 raft_consensus.cc:686] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [term 1 LEADER]: Becoming Leader. State: Replica: d41024d08c864a6da75d14e68e4d4ec7, State: Running, Role: LEADER
I20240506 15:44:24.331521 6215 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:24.331818 6220 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41024d08c864a6da75d14e68e4d4ec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } }
I20240506 15:44:24.332151 6222 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41024d08c864a6da75d14e68e4d4ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41024d08c864a6da75d14e68e4d4ec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } } }
I20240506 15:44:24.332244 6222 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:24.332393 6222 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41024d08c864a6da75d14e68e4d4ec7. Latest consensus state: current_term: 1 leader_uuid: "d41024d08c864a6da75d14e68e4d4ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41024d08c864a6da75d14e68e4d4ec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } } }
I20240506 15:44:24.332444 6222 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:24.332476 6229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:24.332967 6229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:24.334748 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:24.324954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54557 (local address 127.0.0.1:34105)
0506 15:44:24.325128 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:24.325137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:24.325153 (+ 16us) server_negotiation.cc:407] Connection header received
0506 15:44:24.325633 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:24.325647 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:24.325691 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:24.325838 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:24.326601 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:24.327416 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:24.328043 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:24.328682 (+ 639us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:24.330548 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:24.330555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:24.330555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:24.330569 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:24.333123 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:24.333710 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:24.333713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:24.333713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:24.333769 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:24.333976 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:24.333977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:24.333977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:24.334179 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:24.334335 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:24.334471 (+ 136us) server_negotiation.cc:299] Negotiation successful
0506 15:44:24.334546 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20240506 15:44:24.335044 6229 catalog_manager.cc:1347] Generated new cluster ID: f510ed67d1c74c92a1d7aec97100f930
I20240506 15:44:24.335085 6229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:24.335686 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:24.344421 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:24.337174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56387 (local address 127.0.0.1:34105)
0506 15:44:24.337308 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:24.337310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:24.337342 (+ 32us) server_negotiation.cc:407] Connection header received
0506 15:44:24.337419 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:24.337419 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:24.337428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:24.337488 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:24.338279 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:24.339045 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:24.339836 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:24.340018 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:24.341345 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:24.341349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:24.341349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:24.341355 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:24.343048 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:24.343436 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:24.343437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:24.343437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:24.343460 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:24.343633 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:24.343633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:24.343634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:24.343730 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:24.343835 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:24.344236 (+ 401us) server_negotiation.cc:299] Negotiation successful
0506 15:44:24.344298 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20240506 15:44:24.346428 6229 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:24.346922 6229 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:24.352399 6229 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d41024d08c864a6da75d14e68e4d4ec7: Generated new TSK 0
I20240506 15:44:24.352573 6229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:24.357928 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 6147
W20240506 15:44:24.362620 6132 connection.cc:535] client connection to 127.0.0.1:34105 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34105): Connection reset by peer (error 104)
2024-05-06T15:44:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:24 dist-test-slave-v0k1 krb5kdc[6247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:24 dist-test-slave-v0k1 krb5kdc[6247](info): set up 2 sockets
May 06 15:44:24 dist-test-slave-v0k1 krb5kdc[6247](info): commencing operation
krb5kdc: starting...
W20240506 15:44:25.307235 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:25 dist-test-slave-v0k1 krb5kdc[6247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:25.423345 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:25.502919 6260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:25.503158 6260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:25.503183 6260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:25.504573 6260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:25.504630 6260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:25.504645 6260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:25.504657 6260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:25.504671 6260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:25.506392 6260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:25.506623 6260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:25.506817 6260 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:25.511411 6265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:25.512043 6266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:25.512594 6268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:25.513173 6260 server_base.cc:1034] running on GCE node
I20240506 15:44:25.513393 6260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:25.513636 6260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:25.514784 6260 hybrid_clock.cc:648] HybridClock initialized: now 1715010265514767 us; error 37 us; skew 500 ppm
May 06 15:44:25 dist-test-slave-v0k1 krb5kdc[6247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:25.517318 6260 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:25.517724 6260 webserver.cc:460] Webserver started at http://127.0.0.1:41335/ using document root <none> and password file <none>
I20240506 15:44:25.517905 6260 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:25.517951 6260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:25.518050 6260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:25.519093 6260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "7add8f5f2d79426f8192ed91619d52d4"
format_stamp: "Formatted at 2024-05-06 15:44:25 on dist-test-slave-v0k1"
server_key: "db48d86fda33a7b73ad9f9fc798f6278"
server_key_iv: "3f988a9493d7313e1380ea5a632ea94c"
server_key_version: "encryptionkey@0"
I20240506 15:44:25.519408 6260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "7add8f5f2d79426f8192ed91619d52d4"
format_stamp: "Formatted at 2024-05-06 15:44:25 on dist-test-slave-v0k1"
server_key: "db48d86fda33a7b73ad9f9fc798f6278"
server_key_iv: "3f988a9493d7313e1380ea5a632ea94c"
server_key_version: "encryptionkey@0"
I20240506 15:44:25.520599 6260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:44:25.521252 6275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:25.521406 6260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:25.521492 6260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "7add8f5f2d79426f8192ed91619d52d4"
format_stamp: "Formatted at 2024-05-06 15:44:25 on dist-test-slave-v0k1"
server_key: "db48d86fda33a7b73ad9f9fc798f6278"
server_key_iv: "3f988a9493d7313e1380ea5a632ea94c"
server_key_version: "encryptionkey@0"
I20240506 15:44:25.521554 6260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:25.539181 6260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:25.540926 6260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:25.541038 6260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:25.544674 6260 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36125
I20240506 15:44:25.544724 6327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36125 every 8 connection(s)
I20240506 15:44:25.545010 6260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:25.545840 6328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:25.547547 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 6260
I20240506 15:44:25.547670 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:25.547858 18371 external_mini_cluster.cc:1428] Setting key f162f245f0198d9d10f3d3d653a54852
I20240506 15:44:25.548278 6328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4: Bootstrap starting.
I20240506 15:44:25.548862 6328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:25.549131 6328 log.cc:824] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:25.549778 6328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4: No bootstrap required, opened a new log
I20240506 15:44:25.551705 6328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add8f5f2d79426f8192ed91619d52d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } }
I20240506 15:44:25.551857 6328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:25.551891 6328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7add8f5f2d79426f8192ed91619d52d4, State: Initialized, Role: FOLLOWER
I20240506 15:44:25.552011 6328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add8f5f2d79426f8192ed91619d52d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } }
I20240506 15:44:25.552083 6328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:25.552114 6328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:25.552198 6328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:25.552444 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:25.548376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60177 (local address 127.0.0.1:36125)
0506 15:44:25.548556 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:25.548569 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:25.548588 (+ 19us) server_negotiation.cc:407] Connection header received
0506 15:44:25.549023 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:25.549029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:25.549071 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:25.549193 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:25.549583 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:25.550321 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:25.551126 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:25.551280 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:25.551870 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:25.551879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:25.551880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:25.552053 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:25.552074 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:25.552081 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:25.552171 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:25.552272 (+ 101us) server_negotiation.cc:299] Negotiation successful
0506 15:44:25.552299 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20240506 15:44:25.552853 6328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add8f5f2d79426f8192ed91619d52d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } }
I20240506 15:44:25.552971 6328 leader_election.cc:302] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7add8f5f2d79426f8192ed91619d52d4; no voters:
I20240506 15:44:25.553128 6328 leader_election.cc:288] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:25.553175 6333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:25.553355 6328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:25.553475 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:25.553603 6333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [term 1 LEADER]: Becoming Leader. State: Replica: 7add8f5f2d79426f8192ed91619d52d4, State: Running, Role: LEADER
I20240506 15:44:25.553689 6333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add8f5f2d79426f8192ed91619d52d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } }
I20240506 15:44:25.554878 6335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7add8f5f2d79426f8192ed91619d52d4. Latest consensus state: current_term: 1 leader_uuid: "7add8f5f2d79426f8192ed91619d52d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add8f5f2d79426f8192ed91619d52d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } } }
I20240506 15:44:25.554950 6335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:25.554914 6334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7add8f5f2d79426f8192ed91619d52d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add8f5f2d79426f8192ed91619d52d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } } }
I20240506 15:44:25.555061 6334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4 [sys.catalog]: This master's current role is: LEADER
W20240506 15:44:25.555606 6353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:25.555652 6353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:25.555706 6351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:25.555855 6351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:25.557463 6351 catalog_manager.cc:1347] Generated new cluster ID: bc919993fc2841dab7e0db2714a91ed6
I20240506 15:44:25.557526 6351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:25.559543 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:25.555042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56049 (local address 127.0.0.1:36125)
0506 15:44:25.555076 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:25.555077 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:25.555356 (+ 279us) server_negotiation.cc:407] Connection header received
0506 15:44:25.555420 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:25.555422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:25.555431 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:25.555513 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:25.555870 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:25.556490 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:25.557213 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:25.557383 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:25.559254 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:25.559257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:25.559258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:25.559262 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:25.559278 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:25.559283 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:25.559344 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:25.559441 (+ 97us) server_negotiation.cc:299] Negotiation successful
0506 15:44:25.559451 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240506 15:44:25.574040 6351 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:25.574791 6351 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:25.590366 6351 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7add8f5f2d79426f8192ed91619d52d4: Generated new TSK 0
I20240506 15:44:25.590633 6351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:25.597266 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 6260
W20240506 15:44:25.602041 6245 connection.cc:535] client connection to 127.0.0.1:36125 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36125): Connection reset by peer (error 104)
2024-05-06T15:44:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:25 dist-test-slave-v0k1 krb5kdc[6359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:25 dist-test-slave-v0k1 krb5kdc[6359](info): set up 2 sockets
May 06 15:44:25 dist-test-slave-v0k1 krb5kdc[6359](info): commencing operation
krb5kdc: starting...
W20240506 15:44:26.563310 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:26 dist-test-slave-v0k1 krb5kdc[6359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:26.686908 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:26.767416 6372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:26.767609 6372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:26.767632 6372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:26.768982 6372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:26.769028 6372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:26.769042 6372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:26.769054 6372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:26.769065 6372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:26.770676 6372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:26.770893 6372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:26.771077 6372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:26.775753 6378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:26.776121 6380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:26.776677 6377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:26.777431 6372 server_base.cc:1034] running on GCE node
I20240506 15:44:26.777673 6372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:26.777920 6372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:26.779089 6372 hybrid_clock.cc:648] HybridClock initialized: now 1715010266779070 us; error 44 us; skew 500 ppm
May 06 15:44:26 dist-test-slave-v0k1 krb5kdc[6359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:26.781497 6372 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:26.781924 6372 webserver.cc:460] Webserver started at http://127.0.0.1:36477/ using document root <none> and password file <none>
I20240506 15:44:26.782123 6372 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:26.782169 6372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:26.782277 6372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:26.783314 6372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "f85cd918145f46b989bc8b2edf8d23ef"
format_stamp: "Formatted at 2024-05-06 15:44:26 on dist-test-slave-v0k1"
server_key: "a8d3dc85d05311be9d251a7098d6daca"
server_key_iv: "1cdeec352b8358fe3949291d0faae191"
server_key_version: "encryptionkey@0"
I20240506 15:44:26.783632 6372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "f85cd918145f46b989bc8b2edf8d23ef"
format_stamp: "Formatted at 2024-05-06 15:44:26 on dist-test-slave-v0k1"
server_key: "a8d3dc85d05311be9d251a7098d6daca"
server_key_iv: "1cdeec352b8358fe3949291d0faae191"
server_key_version: "encryptionkey@0"
I20240506 15:44:26.784849 6372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:44:26.785563 6387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:26.785760 6372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:26.785840 6372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "f85cd918145f46b989bc8b2edf8d23ef"
format_stamp: "Formatted at 2024-05-06 15:44:26 on dist-test-slave-v0k1"
server_key: "a8d3dc85d05311be9d251a7098d6daca"
server_key_iv: "1cdeec352b8358fe3949291d0faae191"
server_key_version: "encryptionkey@0"
I20240506 15:44:26.785902 6372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:26.789469 6372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:26.791424 6372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:26.791540 6372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:26.795373 6372 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36245
I20240506 15:44:26.795416 6439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36245 every 8 connection(s)
I20240506 15:44:26.795706 6372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:26.796525 6440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:26.798601 6440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef: Bootstrap starting.
I20240506 15:44:26.799187 6440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:26.799432 6440 log.cc:824] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:26.800062 6440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef: No bootstrap required, opened a new log
I20240506 15:44:26.801301 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 6372
I20240506 15:44:26.801396 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:26.801581 18371 external_mini_cluster.cc:1428] Setting key 82f9f6affa793b94b70f305ab2fcf0e0
I20240506 15:44:26.801553 6440 raft_consensus.cc:348] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cd918145f46b989bc8b2edf8d23ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20240506 15:44:26.801684 6440 raft_consensus.cc:374] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:26.801723 6440 raft_consensus.cc:729] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85cd918145f46b989bc8b2edf8d23ef, State: Initialized, Role: FOLLOWER
I20240506 15:44:26.801844 6440 consensus_queue.cc:260] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cd918145f46b989bc8b2edf8d23ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20240506 15:44:26.801918 6440 raft_consensus.cc:388] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:26.801971 6440 raft_consensus.cc:482] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:26.802018 6440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:26.802558 6440 raft_consensus.cc:504] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cd918145f46b989bc8b2edf8d23ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20240506 15:44:26.802654 6440 leader_election.cc:302] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f85cd918145f46b989bc8b2edf8d23ef; no voters:
I20240506 15:44:26.802805 6440 leader_election.cc:288] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:26.802978 6445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:26.803094 6440 sys_catalog.cc:564] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:26.803175 6445 raft_consensus.cc:686] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [term 1 LEADER]: Becoming Leader. State: Replica: f85cd918145f46b989bc8b2edf8d23ef, State: Running, Role: LEADER
I20240506 15:44:26.803329 6445 consensus_queue.cc:237] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cd918145f46b989bc8b2edf8d23ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20240506 15:44:26.803680 6446 sys_catalog.cc:455] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f85cd918145f46b989bc8b2edf8d23ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cd918145f46b989bc8b2edf8d23ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } } }
I20240506 15:44:26.803787 6446 sys_catalog.cc:458] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:26.803829 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:26.801972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38389 (local address 127.0.0.1:36245)
0506 15:44:26.802167 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:26.802180 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:26.802265 (+ 85us) server_negotiation.cc:407] Connection header received
0506 15:44:26.802401 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:26.802406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:26.802428 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:26.802544 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:26.803317 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:26.803320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:26.803320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:26.803325 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:26.803347 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:26.803356 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:26.803406 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:26.803534 (+ 128us) server_negotiation.cc:299] Negotiation successful
0506 15:44:26.803569 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20240506 15:44:26.803671 6445 sys_catalog.cc:455] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader f85cd918145f46b989bc8b2edf8d23ef. Latest consensus state: current_term: 1 leader_uuid: "f85cd918145f46b989bc8b2edf8d23ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cd918145f46b989bc8b2edf8d23ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } } }
I20240506 15:44:26.803992 6445 sys_catalog.cc:458] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:26.804821 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20240506 15:44:26.804972 6460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:26.805016 6460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:26.805066 6454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:26.805202 6454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:26.806871 6454 catalog_manager.cc:1347] Generated new cluster ID: 9ab8e166e9264e42beeb2fdecae21f11
I20240506 15:44:26.806926 6454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:26.807484 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:26.806120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36759 (local address 127.0.0.1:36245)
0506 15:44:26.806161 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:26.806163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:26.806414 (+ 251us) server_negotiation.cc:407] Connection header received
0506 15:44:26.806471 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:26.806472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:26.806477 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:26.806537 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:26.807264 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:26.807265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:26.807265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:26.807269 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:26.807283 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:26.807288 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:26.807343 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:26.807411 (+ 68us) server_negotiation.cc:299] Negotiation successful
0506 15:44:26.807423 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:44:26.817766 6454 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:26.817914 6454 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:26.824816 6454 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f85cd918145f46b989bc8b2edf8d23ef: Generated new TSK 0
I20240506 15:44:26.825019 6454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:26.838483 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 6372
2024-05-06T15:44:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1241 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:26 dist-test-slave-v0k1 krb5kdc[6471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:26 dist-test-slave-v0k1 krb5kdc[6471](info): set up 2 sockets
May 06 15:44:26 dist-test-slave-v0k1 krb5kdc[6471](info): commencing operation
krb5kdc: starting...
W20240506 15:44:27.799335 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:27 dist-test-slave-v0k1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:27.917376 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:27.997381 6484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:27.997597 6484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:27.997635 6484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:27.998981 6484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:27.999058 6484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:27.999079 6484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:27.999100 6484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:27.999120 6484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:28.000797 6484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:28.001037 6484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:28.001225 6484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:28.005865 6489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:28.006323 6492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:28.006906 6490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:28.007728 6484 server_base.cc:1034] running on GCE node
I20240506 15:44:28.007951 6484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:28.008199 6484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:28.009337 6484 hybrid_clock.cc:648] HybridClock initialized: now 1715010268009318 us; error 32 us; skew 500 ppm
May 06 15:44:28 dist-test-slave-v0k1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:28.011818 6484 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:28.012238 6484 webserver.cc:460] Webserver started at http://127.0.0.1:45163/ using document root <none> and password file <none>
I20240506 15:44:28.012418 6484 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:28.012457 6484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:28.012575 6484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:28.013778 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "7f630db90e50497fab664fb14d2a605c"
format_stamp: "Formatted at 2024-05-06 15:44:28 on dist-test-slave-v0k1"
server_key: "75b47ccc62cefb1e2d1c7ac0a9ba3481"
server_key_iv: "c43ec722327f35d81742dceb296b59f7"
server_key_version: "encryptionkey@0"
I20240506 15:44:28.014110 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "7f630db90e50497fab664fb14d2a605c"
format_stamp: "Formatted at 2024-05-06 15:44:28 on dist-test-slave-v0k1"
server_key: "75b47ccc62cefb1e2d1c7ac0a9ba3481"
server_key_iv: "c43ec722327f35d81742dceb296b59f7"
server_key_version: "encryptionkey@0"
I20240506 15:44:28.015424 6484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:44:28.016109 6499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:28.016269 6484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:28.016338 6484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "7f630db90e50497fab664fb14d2a605c"
format_stamp: "Formatted at 2024-05-06 15:44:28 on dist-test-slave-v0k1"
server_key: "75b47ccc62cefb1e2d1c7ac0a9ba3481"
server_key_iv: "c43ec722327f35d81742dceb296b59f7"
server_key_version: "encryptionkey@0"
I20240506 15:44:28.016408 6484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:28.040035 6484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:28.041828 6484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:28.041973 6484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:28.045835 6484 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34789
I20240506 15:44:28.045869 6551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34789 every 8 connection(s)
I20240506 15:44:28.046190 6484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:28.047113 6552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:28.049286 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c: Bootstrap starting.
I20240506 15:44:28.049882 6552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:28.050139 6552 log.cc:824] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:28.050832 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c: No bootstrap required, opened a new log
I20240506 15:44:28.052127 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 6484
I20240506 15:44:28.052229 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:28.052388 18371 external_mini_cluster.cc:1428] Setting key 5f9e56e648e4d134073650ea83901eab
I20240506 15:44:28.052446 6552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f630db90e50497fab664fb14d2a605c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34789 } }
I20240506 15:44:28.052616 6552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:28.052651 6552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f630db90e50497fab664fb14d2a605c, State: Initialized, Role: FOLLOWER
I20240506 15:44:28.052831 6552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f630db90e50497fab664fb14d2a605c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34789 } }
I20240506 15:44:28.052994 6552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:28.053043 6552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:28.053094 6552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:28.053686 6552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f630db90e50497fab664fb14d2a605c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34789 } }
I20240506 15:44:28.053790 6552 leader_election.cc:302] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f630db90e50497fab664fb14d2a605c; no voters:
I20240506 15:44:28.053946 6552 leader_election.cc:288] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:28.054172 6557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:28.054210 6552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:28.054303 6557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [term 1 LEADER]: Becoming Leader. State: Replica: 7f630db90e50497fab664fb14d2a605c, State: Running, Role: LEADER
I20240506 15:44:28.054399 6557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f630db90e50497fab664fb14d2a605c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34789 } }
I20240506 15:44:28.054718 6557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f630db90e50497fab664fb14d2a605c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f630db90e50497fab664fb14d2a605c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34789 } } }
I20240506 15:44:28.054867 6557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:28.055050 6559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f630db90e50497fab664fb14d2a605c. Latest consensus state: current_term: 1 leader_uuid: "7f630db90e50497fab664fb14d2a605c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f630db90e50497fab664fb14d2a605c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34789 } } }
I20240506 15:44:28.055136 6564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:28.055689 6564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:28.055213 6559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:28.058367 6564 catalog_manager.cc:1347] Generated new cluster ID: ea24febce53845468a1e14c799066373
I20240506 15:44:28.058430 6564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:44:28 dist-test-slave-v0k1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240506 15:44:28.063660 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:28.052899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40891 (local address 127.0.0.1:34789)
0506 15:44:28.053141 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:28.053154 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:28.053177 (+ 23us) server_negotiation.cc:407] Connection header received
0506 15:44:28.053277 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:28.053281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:28.053359 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:28.053495 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:28.054652 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:28.056235 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:28.057123 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:28.057405 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:28.059774 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:28.059804 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:28.059804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:28.059825 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:28.062065 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:28.062474 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:28.062477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:28.062478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:28.062545 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:28.062699 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:28.062700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:28.062700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:28.062964 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:28.063169 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:28.063290 (+ 121us) server_negotiation.cc:299] Negotiation successful
0506 15:44:28.063451 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":122,"threads_started":1}
I20240506 15:44:28.065172 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:28.073597 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:28.066572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43071 (local address 127.0.0.1:34789)
0506 15:44:28.066609 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:28.066611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:28.066747 (+ 136us) server_negotiation.cc:407] Connection header received
0506 15:44:28.066797 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:28.066798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:28.066812 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:28.066883 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:28.067681 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:28.068416 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:28.069128 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:28.069330 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:28.070599 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:28.070604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:28.070604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:28.070612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:28.072529 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:28.072911 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:28.072912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:28.072912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:28.072937 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:28.073080 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:28.073081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:28.073081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:28.073195 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:28.073305 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:28.073403 (+ 98us) server_negotiation.cc:299] Negotiation successful
0506 15:44:28.073494 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240506 15:44:28.080802 6564 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:28.081593 6564 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:28.087261 6564 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7f630db90e50497fab664fb14d2a605c: Generated new TSK 0
I20240506 15:44:28.087458 6564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:28.094061 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 6484
2024-05-06T15:44:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1255 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:28 dist-test-slave-v0k1 krb5kdc[6583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:28 dist-test-slave-v0k1 krb5kdc[6583](info): set up 2 sockets
May 06 15:44:28 dist-test-slave-v0k1 krb5kdc[6583](info): commencing operation
krb5kdc: starting...
W20240506 15:44:29.067200 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:29 dist-test-slave-v0k1 krb5kdc[6583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:29.186641 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:29.266134 6596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:29.266340 6596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:29.266368 6596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:29.267763 6596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:29.267812 6596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:29.267833 6596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:29.267853 6596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:29.267871 6596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:29.269519 6596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:29.269768 6596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:29.269953 6596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:29.274703 6602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:29.275233 6604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:29.275682 6601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:29.276574 6596 server_base.cc:1034] running on GCE node
I20240506 15:44:29.276799 6596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:29.277035 6596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:29.278173 6596 hybrid_clock.cc:648] HybridClock initialized: now 1715010269278156 us; error 30 us; skew 500 ppm
May 06 15:44:29 dist-test-slave-v0k1 krb5kdc[6583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:29.280709 6596 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:29.281129 6596 webserver.cc:460] Webserver started at http://127.0.0.1:32831/ using document root <none> and password file <none>
I20240506 15:44:29.281320 6596 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:29.281364 6596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:29.281497 6596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:29.282483 6596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "80d5fc408a4840deb61a9166797121b2"
format_stamp: "Formatted at 2024-05-06 15:44:29 on dist-test-slave-v0k1"
server_key: "38d0e563153da16b4bfd13486df78d31"
server_key_iv: "6e97588c08c827c472907fe51cd75f7f"
server_key_version: "encryptionkey@0"
I20240506 15:44:29.282788 6596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "80d5fc408a4840deb61a9166797121b2"
format_stamp: "Formatted at 2024-05-06 15:44:29 on dist-test-slave-v0k1"
server_key: "38d0e563153da16b4bfd13486df78d31"
server_key_iv: "6e97588c08c827c472907fe51cd75f7f"
server_key_version: "encryptionkey@0"
I20240506 15:44:29.284005 6596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:44:29.284651 6611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:29.284802 6596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:29.284878 6596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "80d5fc408a4840deb61a9166797121b2"
format_stamp: "Formatted at 2024-05-06 15:44:29 on dist-test-slave-v0k1"
server_key: "38d0e563153da16b4bfd13486df78d31"
server_key_iv: "6e97588c08c827c472907fe51cd75f7f"
server_key_version: "encryptionkey@0"
I20240506 15:44:29.284937 6596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:29.300501 6596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:29.302632 6596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:29.302757 6596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:29.306671 6596 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37995
I20240506 15:44:29.306726 6663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37995 every 8 connection(s)
I20240506 15:44:29.307080 6596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:29.307931 6664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:29.310166 6664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2: Bootstrap starting.
I20240506 15:44:29.310779 6664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:29.310935 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 6596
I20240506 15:44:29.311076 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:29.311153 6664 log.cc:824] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:29.311272 18371 external_mini_cluster.cc:1428] Setting key 12facf493f178b4161d7396247dda71b
I20240506 15:44:29.311957 6664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2: No bootstrap required, opened a new log
I20240506 15:44:29.313977 6664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d5fc408a4840deb61a9166797121b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37995 } }
I20240506 15:44:29.314105 6664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:29.314132 6664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d5fc408a4840deb61a9166797121b2, State: Initialized, Role: FOLLOWER
I20240506 15:44:29.314246 6664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d5fc408a4840deb61a9166797121b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37995 } }
I20240506 15:44:29.314309 6664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:29.314342 6664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:29.314394 6664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:29.314953 6664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d5fc408a4840deb61a9166797121b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37995 } }
I20240506 15:44:29.315097 6664 leader_election.cc:302] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80d5fc408a4840deb61a9166797121b2; no voters:
I20240506 15:44:29.315271 6664 leader_election.cc:288] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:29.315358 6669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:29.315541 6664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:29.315582 6669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [term 1 LEADER]: Becoming Leader. State: Replica: 80d5fc408a4840deb61a9166797121b2, State: Running, Role: LEADER
I20240506 15:44:29.315750 6669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d5fc408a4840deb61a9166797121b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37995 } }
I20240506 15:44:29.316166 6670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80d5fc408a4840deb61a9166797121b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d5fc408a4840deb61a9166797121b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37995 } } }
I20240506 15:44:29.316243 6670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:29.316188 6671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80d5fc408a4840deb61a9166797121b2. Latest consensus state: current_term: 1 leader_uuid: "80d5fc408a4840deb61a9166797121b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d5fc408a4840deb61a9166797121b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37995 } } }
I20240506 15:44:29.316283 6671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:29.316454 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:29.311887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35987 (local address 127.0.0.1:37995)
0506 15:44:29.312075 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:29.312087 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:29.312110 (+ 23us) server_negotiation.cc:407] Connection header received
0506 15:44:29.312605 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:29.312611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:29.312655 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:29.312790 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:29.313328 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:29.314082 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:29.314776 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:29.315061 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:29.315574 (+ 513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:29.315583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:29.315583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:29.315771 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:29.315798 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:29.315806 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:29.316160 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:29.316200 (+ 40us) server_negotiation.cc:299] Negotiation successful
0506 15:44:29.316246 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20240506 15:44:29.316555 6678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:29.317113 6678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:29.317771 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:29.318806 6678 catalog_manager.cc:1347] Generated new cluster ID: 047f2a328512467cb4e5141f250f44b0
I20240506 15:44:29.318864 6678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:29.321890 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:29.319374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53063 (local address 127.0.0.1:37995)
0506 15:44:29.319414 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:29.319416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:29.319630 (+ 214us) server_negotiation.cc:407] Connection header received
0506 15:44:29.319690 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:29.319691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:29.319701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:29.319762 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:29.320026 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:29.320460 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:29.320962 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:29.321178 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:29.321639 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:29.321643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:29.321643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:29.321648 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:29.321665 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:29.321669 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:29.321716 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:29.321778 (+ 62us) server_negotiation.cc:299] Negotiation successful
0506 15:44:29.321806 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240506 15:44:29.329180 6678 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:29.329718 6678 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:29.334578 6678 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 80d5fc408a4840deb61a9166797121b2: Generated new TSK 0
I20240506 15:44:29.334764 6678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:29.337550 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 6596
2024-05-06T15:44:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:29 dist-test-slave-v0k1 krb5kdc[6695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:29 dist-test-slave-v0k1 krb5kdc[6695](info): set up 2 sockets
May 06 15:44:29 dist-test-slave-v0k1 krb5kdc[6695](info): commencing operation
krb5kdc: starting...
W20240506 15:44:30.283742 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:30 dist-test-slave-v0k1 krb5kdc[6695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:30.397874 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:30.479416 6708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:30.479622 6708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:30.479646 6708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:30.480971 6708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:30.481019 6708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:30.481034 6708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:30.481048 6708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:30.481094 6708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:30.482714 6708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:30.482928 6708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:30.483143 6708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:30.487705 6716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:30.488390 6713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:30.488965 6714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:30.489519 6708 server_base.cc:1034] running on GCE node
I20240506 15:44:30.489702 6708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:30.489908 6708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:30.491082 6708 hybrid_clock.cc:648] HybridClock initialized: now 1715010270491050 us; error 42 us; skew 500 ppm
May 06 15:44:30 dist-test-slave-v0k1 krb5kdc[6695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:30.493503 6708 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:30.493898 6708 webserver.cc:460] Webserver started at http://127.0.0.1:41443/ using document root <none> and password file <none>
I20240506 15:44:30.494088 6708 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:30.494135 6708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:30.494243 6708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:30.495262 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "d87a3893af5e4120814710c84518ef48"
format_stamp: "Formatted at 2024-05-06 15:44:30 on dist-test-slave-v0k1"
server_key: "53ccca1b23601881f198c57daa399d32"
server_key_iv: "4348afaf87c9474b47bb3421e8325d2c"
server_key_version: "encryptionkey@0"
I20240506 15:44:30.495570 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "d87a3893af5e4120814710c84518ef48"
format_stamp: "Formatted at 2024-05-06 15:44:30 on dist-test-slave-v0k1"
server_key: "53ccca1b23601881f198c57daa399d32"
server_key_iv: "4348afaf87c9474b47bb3421e8325d2c"
server_key_version: "encryptionkey@0"
I20240506 15:44:30.496717 6708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:30.497330 6723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:30.497489 6708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:30.497566 6708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "d87a3893af5e4120814710c84518ef48"
format_stamp: "Formatted at 2024-05-06 15:44:30 on dist-test-slave-v0k1"
server_key: "53ccca1b23601881f198c57daa399d32"
server_key_iv: "4348afaf87c9474b47bb3421e8325d2c"
server_key_version: "encryptionkey@0"
I20240506 15:44:30.497627 6708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:30.501112 6708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:30.502863 6708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:30.502979 6708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:30.506485 6708 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35277
I20240506 15:44:30.506531 6775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35277 every 8 connection(s)
I20240506 15:44:30.506807 6708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:30.507624 6776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:30.509651 6776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48: Bootstrap starting.
I20240506 15:44:30.510181 6776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:30.510416 6776 log.cc:824] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:30.510993 6776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48: No bootstrap required, opened a new log
I20240506 15:44:30.511852 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 6708
I20240506 15:44:30.511981 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:30.512151 18371 external_mini_cluster.cc:1428] Setting key 79e6e031094a32abdbb2ef578013b718
I20240506 15:44:30.512706 6776 raft_consensus.cc:348] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87a3893af5e4120814710c84518ef48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } }
I20240506 15:44:30.512866 6776 raft_consensus.cc:374] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:30.512892 6776 raft_consensus.cc:729] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87a3893af5e4120814710c84518ef48, State: Initialized, Role: FOLLOWER
I20240506 15:44:30.512988 6776 consensus_queue.cc:260] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87a3893af5e4120814710c84518ef48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } }
I20240506 15:44:30.513062 6776 raft_consensus.cc:388] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:30.513144 6776 raft_consensus.cc:482] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:30.513196 6776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:30.514305 6776 raft_consensus.cc:504] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87a3893af5e4120814710c84518ef48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } }
I20240506 15:44:30.514415 6776 leader_election.cc:302] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d87a3893af5e4120814710c84518ef48; no voters:
I20240506 15:44:30.514420 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:30.512621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43703 (local address 127.0.0.1:35277)
0506 15:44:30.512809 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:30.512820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:30.512843 (+ 23us) server_negotiation.cc:407] Connection header received
0506 15:44:30.512940 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:30.512946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:30.512981 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:30.513124 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:30.513758 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:30.513762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:30.513762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:30.513766 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:30.513791 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:30.513798 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:30.513882 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:30.513918 (+ 36us) server_negotiation.cc:299] Negotiation successful
0506 15:44:30.513950 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20240506 15:44:30.514605 6776 leader_election.cc:288] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:30.514647 6781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:30.514822 6781 raft_consensus.cc:686] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [term 1 LEADER]: Becoming Leader. State: Replica: d87a3893af5e4120814710c84518ef48, State: Running, Role: LEADER
I20240506 15:44:30.514827 6776 sys_catalog.cc:564] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:30.514933 6781 consensus_queue.cc:237] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87a3893af5e4120814710c84518ef48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } }
I20240506 15:44:30.515413 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:30.516278 6782 sys_catalog.cc:455] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d87a3893af5e4120814710c84518ef48. Latest consensus state: current_term: 1 leader_uuid: "d87a3893af5e4120814710c84518ef48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87a3893af5e4120814710c84518ef48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } } }
I20240506 15:44:30.516381 6782 sys_catalog.cc:458] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:30.516516 6783 sys_catalog.cc:455] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d87a3893af5e4120814710c84518ef48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87a3893af5e4120814710c84518ef48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } } }
I20240506 15:44:30.516863 6783 sys_catalog.cc:458] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48 [sys.catalog]: This master's current role is: LEADER
W20240506 15:44:30.517112 6797 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:30.517204 6797 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:30.517277 6799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:30.517455 6799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:30.518107 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:30.516843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57755 (local address 127.0.0.1:35277)
0506 15:44:30.516877 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:30.516878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:30.517132 (+ 254us) server_negotiation.cc:407] Connection header received
0506 15:44:30.517194 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:30.517195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:30.517200 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:30.517259 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:30.517870 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:30.517871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:30.517872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:30.517876 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:30.517892 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:30.517896 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:30.517967 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:30.518036 (+ 69us) server_negotiation.cc:299] Negotiation successful
0506 15:44:30.518049 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240506 15:44:30.518754 6799 catalog_manager.cc:1347] Generated new cluster ID: ced665fbcf0646ce93bb41b8a422de3b
I20240506 15:44:30.518805 6799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:30.523068 6799 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:30.523214 6799 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:30.535533 6799 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d87a3893af5e4120814710c84518ef48: Generated new TSK 0
I20240506 15:44:30.535774 6799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:30.540136 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 6708
2024-05-06T15:44:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:30 dist-test-slave-v0k1 krb5kdc[6807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:30 dist-test-slave-v0k1 krb5kdc[6807](info): set up 2 sockets
May 06 15:44:30 dist-test-slave-v0k1 krb5kdc[6807](info): commencing operation
krb5kdc: starting...
W20240506 15:44:31.503259 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:31 dist-test-slave-v0k1 krb5kdc[6807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:31.621892 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:31.701714 6820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:31.701930 6820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:31.701954 6820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:31.703341 6820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:31.703393 6820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:31.703408 6820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:31.703419 6820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:31.703431 6820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:31.705109 6820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:31.705322 6820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:31.705514 6820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:31.710558 6828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:31.711234 6825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:31.711967 6826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:31.712531 6820 server_base.cc:1034] running on GCE node
I20240506 15:44:31.712756 6820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:31.713007 6820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:31.714139 6820 hybrid_clock.cc:648] HybridClock initialized: now 1715010271714124 us; error 31 us; skew 500 ppm
May 06 15:44:31 dist-test-slave-v0k1 krb5kdc[6807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:31.716585 6820 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:31.717062 6820 webserver.cc:460] Webserver started at http://127.0.0.1:38331/ using document root <none> and password file <none>
I20240506 15:44:31.717242 6820 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:31.717288 6820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:31.717396 6820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:31.718384 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "af15e29858814df5818e0659ac4049a2"
format_stamp: "Formatted at 2024-05-06 15:44:31 on dist-test-slave-v0k1"
server_key: "5866b69a87431637a86b70c063232f78"
server_key_iv: "aa71fcf16cd526ea58c95f59d9303323"
server_key_version: "encryptionkey@0"
I20240506 15:44:31.718729 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "af15e29858814df5818e0659ac4049a2"
format_stamp: "Formatted at 2024-05-06 15:44:31 on dist-test-slave-v0k1"
server_key: "5866b69a87431637a86b70c063232f78"
server_key_iv: "aa71fcf16cd526ea58c95f59d9303323"
server_key_version: "encryptionkey@0"
I20240506 15:44:31.719974 6820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:44:31.720602 6835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:31.720752 6820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:31.720826 6820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "af15e29858814df5818e0659ac4049a2"
format_stamp: "Formatted at 2024-05-06 15:44:31 on dist-test-slave-v0k1"
server_key: "5866b69a87431637a86b70c063232f78"
server_key_iv: "aa71fcf16cd526ea58c95f59d9303323"
server_key_version: "encryptionkey@0"
I20240506 15:44:31.720888 6820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:31.730562 6820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:31.732707 6820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:31.732836 6820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:31.736725 6820 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42787
I20240506 15:44:31.736760 6887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42787 every 8 connection(s)
I20240506 15:44:31.737095 6820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:31.737942 6888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:31.740077 6888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2: Bootstrap starting.
I20240506 15:44:31.740623 6888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:31.740856 6888 log.cc:824] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:31.741497 6888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2: No bootstrap required, opened a new log
I20240506 15:44:31.742985 6888 raft_consensus.cc:348] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af15e29858814df5818e0659ac4049a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20240506 15:44:31.743150 6888 raft_consensus.cc:374] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:31.743171 6888 raft_consensus.cc:729] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af15e29858814df5818e0659ac4049a2, State: Initialized, Role: FOLLOWER
I20240506 15:44:31.743314 6888 consensus_queue.cc:260] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af15e29858814df5818e0659ac4049a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20240506 15:44:31.743386 6888 raft_consensus.cc:388] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:31.743422 6888 raft_consensus.cc:482] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:31.743471 6888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:31.743973 6888 raft_consensus.cc:504] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af15e29858814df5818e0659ac4049a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20240506 15:44:31.744060 6888 leader_election.cc:302] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af15e29858814df5818e0659ac4049a2; no voters:
I20240506 15:44:31.744210 6888 leader_election.cc:288] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:31.744263 6891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:31.744421 6888 sys_catalog.cc:564] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:31.744704 6891 raft_consensus.cc:686] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [term 1 LEADER]: Becoming Leader. State: Replica: af15e29858814df5818e0659ac4049a2, State: Running, Role: LEADER
I20240506 15:44:31.744820 6891 consensus_queue.cc:237] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af15e29858814df5818e0659ac4049a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20240506 15:44:31.745167 6892 sys_catalog.cc:455] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af15e29858814df5818e0659ac4049a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af15e29858814df5818e0659ac4049a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } } }
I20240506 15:44:31.745194 6893 sys_catalog.cc:455] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af15e29858814df5818e0659ac4049a2. Latest consensus state: current_term: 1 leader_uuid: "af15e29858814df5818e0659ac4049a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af15e29858814df5818e0659ac4049a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } } }
I20240506 15:44:31.745254 6892 sys_catalog.cc:458] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:31.745260 6893 sys_catalog.cc:458] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:31.745425 6900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:31.745918 6900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:31.746675 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 6820
I20240506 15:44:31.746783 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:31.746960 18371 external_mini_cluster.cc:1428] Setting key 724c9cb0ad693c1d82415aea49090552
I20240506 15:44:31.748785 6900 catalog_manager.cc:1347] Generated new cluster ID: ab8c2d8db72547969bf221556423c2df
I20240506 15:44:31.748885 6900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 06 15:44:31 dist-test-slave-v0k1 krb5kdc[6807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240506 15:44:31.754618 6900 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:31.755178 6900 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:31.757400 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:31.747411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:52691 (local address 127.0.0.1:42787)
0506 15:44:31.747674 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:31.747687 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:31.747713 (+ 26us) server_negotiation.cc:407] Connection header received
0506 15:44:31.747808 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:31.747812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:31.747850 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:31.748051 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:31.748759 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:31.749882 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:31.750719 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:31.750934 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:31.753578 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:31.753584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:31.753585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:31.753606 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:31.755842 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:31.756299 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:31.756302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:31.756302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:31.756366 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:31.756564 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:31.756565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:31.756565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:31.756789 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:31.756949 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:31.757067 (+ 118us) server_negotiation.cc:299] Negotiation successful
0506 15:44:31.757179 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":88,"threads_started":1}
I20240506 15:44:31.758457 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:31.762450 6900 catalog_manager.cc:5857] T 00000000000000000000000000000000 P af15e29858814df5818e0659ac4049a2: Generated new TSK 0
I20240506 15:44:31.762670 6900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:31.766211 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:31.759919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:41527 (local address 127.0.0.1:42787)
0506 15:44:31.759962 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:31.759963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:31.760030 (+ 67us) server_negotiation.cc:407] Connection header received
0506 15:44:31.760088 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:31.760089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:31.760100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:31.760167 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:31.760907 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:31.761705 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:31.762400 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:31.762581 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:31.763745 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:31.763748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:31.763748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:31.763754 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:31.765279 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:31.765607 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:31.765608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:31.765608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:31.765631 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:31.765763 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:31.765764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:31.765764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:31.765861 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:31.765964 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:31.766055 (+ 91us) server_negotiation.cc:299] Negotiation successful
0506 15:44:31.766122 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:44:31.768431 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 6820
W20240506 15:44:31.773114 6805 connection.cc:535] client connection to 127.0.0.1:42787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42787): Connection reset by peer (error 104)
2024-05-06T15:44:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:31 dist-test-slave-v0k1 krb5kdc[6919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:31 dist-test-slave-v0k1 krb5kdc[6919](info): set up 2 sockets
May 06 15:44:31 dist-test-slave-v0k1 krb5kdc[6919](info): commencing operation
krb5kdc: starting...
W20240506 15:44:32.711360 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:32 dist-test-slave-v0k1 krb5kdc[6919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:32.826870 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:32.904778 6932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:32.904976 6932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:32.904999 6932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:32.906239 6932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:32.906276 6932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:32.906289 6932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:32.906301 6932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:32.906312 6932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:32.907907 6932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:32.908079 6932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:32.908216 6932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:32.912878 6937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:32.913471 6940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:32.914052 6938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:32.914877 6932 server_base.cc:1034] running on GCE node
I20240506 15:44:32.915104 6932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:32.915313 6932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:32.916440 6932 hybrid_clock.cc:648] HybridClock initialized: now 1715010272916427 us; error 29 us; skew 500 ppm
May 06 15:44:32 dist-test-slave-v0k1 krb5kdc[6919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:32.924791 6932 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:32.925227 6932 webserver.cc:460] Webserver started at http://127.0.0.1:43539/ using document root <none> and password file <none>
I20240506 15:44:32.925406 6932 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:32.925447 6932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:32.925544 6932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:32.926532 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd"
format_stamp: "Formatted at 2024-05-06 15:44:32 on dist-test-slave-v0k1"
server_key: "138cd5e279eb14c7ff30d3c4bfbcef7f"
server_key_iv: "cd07e30b659340b6d933a0aa7bd9ab03"
server_key_version: "encryptionkey@0"
I20240506 15:44:32.926826 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd"
format_stamp: "Formatted at 2024-05-06 15:44:32 on dist-test-slave-v0k1"
server_key: "138cd5e279eb14c7ff30d3c4bfbcef7f"
server_key_iv: "cd07e30b659340b6d933a0aa7bd9ab03"
server_key_version: "encryptionkey@0"
I20240506 15:44:32.928061 6932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:44:32.928740 6947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:32.928894 6932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:32.928974 6932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd"
format_stamp: "Formatted at 2024-05-06 15:44:32 on dist-test-slave-v0k1"
server_key: "138cd5e279eb14c7ff30d3c4bfbcef7f"
server_key_iv: "cd07e30b659340b6d933a0aa7bd9ab03"
server_key_version: "encryptionkey@0"
I20240506 15:44:32.929039 6932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:32.945154 6932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:32.947227 6932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:32.947345 6932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:32.950929 6932 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34543
I20240506 15:44:32.950978 6999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34543 every 8 connection(s)
I20240506 15:44:32.951308 6932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:32.952095 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:32.954128 7000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd: Bootstrap starting.
I20240506 15:44:32.954661 7000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:32.954900 7000 log.cc:824] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:32.955543 7000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd: No bootstrap required, opened a new log
I20240506 15:44:32.956926 7000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } }
I20240506 15:44:32.957065 7000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:32.957091 7000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c73a3c08ccc4f8798bb93623c7f5cdd, State: Initialized, Role: FOLLOWER
I20240506 15:44:32.957196 7000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } }
I20240506 15:44:32.957252 7000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:32.957283 7000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:32.957330 7000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:32.957883 7000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } }
I20240506 15:44:32.957985 7000 leader_election.cc:302] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c73a3c08ccc4f8798bb93623c7f5cdd; no voters:
I20240506 15:44:32.958235 7000 leader_election.cc:288] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:32.958281 7003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:32.958483 7000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:32.958685 7003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [term 1 LEADER]: Becoming Leader. State: Replica: 0c73a3c08ccc4f8798bb93623c7f5cdd, State: Running, Role: LEADER
I20240506 15:44:32.958771 7003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } }
I20240506 15:44:32.959124 7005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c73a3c08ccc4f8798bb93623c7f5cdd. Latest consensus state: current_term: 1 leader_uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } } }
I20240506 15:44:32.959142 7004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c73a3c08ccc4f8798bb93623c7f5cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } } }
I20240506 15:44:32.959235 7005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:32.959249 7004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:32.959429 7012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:32.959977 7012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:32.961148 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 6932
I20240506 15:44:32.961261 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:32.961433 7012 catalog_manager.cc:1347] Generated new cluster ID: 2bc79d95ba574cc2bd30489422405b89
I20240506 15:44:32.961480 18371 external_mini_cluster.cc:1428] Setting key 39a6ffc853c13eedd51af9ee9596c555
I20240506 15:44:32.961498 7012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:32.966573 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:32.961916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:52415 (local address 127.0.0.1:34543)
0506 15:44:32.963321 (+ 1405us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:32.963330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:32.963346 (+ 16us) server_negotiation.cc:407] Connection header received
0506 15:44:32.963425 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:32.963430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:32.963474 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:32.963580 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:32.963967 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:32.964681 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:32.965259 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:32.965503 (+ 244us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:32.965960 (+ 457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:32.965967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:32.965968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:32.966137 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:32.966160 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:32.966167 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:32.966254 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:32.966395 (+ 141us) server_negotiation.cc:299] Negotiation successful
0506 15:44:32.966425 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1369,"thread_start_us":72,"threads_started":1}
I20240506 15:44:32.967482 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:32.973438 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:32.968706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:55675 (local address 127.0.0.1:34543)
0506 15:44:32.968745 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:32.968746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:32.971197 (+ 2451us) server_negotiation.cc:407] Connection header received
0506 15:44:32.971256 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:32.971257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:32.971270 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:32.971329 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:32.971582 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:32.972010 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:32.972500 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:32.972681 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:32.973115 (+ 434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:32.973118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:32.973118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:32.973122 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:32.973137 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:32.973141 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:32.973206 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:32.973291 (+ 85us) server_negotiation.cc:299] Negotiation successful
0506 15:44:32.973302 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:44:32.985571 7012 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:32.986287 7012 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:32.995318 7012 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0c73a3c08ccc4f8798bb93623c7f5cdd: Generated new TSK 0
I20240506 15:44:32.995499 7012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:33.001876 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 6932
W20240506 15:44:33.006364 6917 connection.cc:535] client connection to 127.0.0.1:34543 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34543): Connection reset by peer (error 104)
2024-05-06T15:44:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:33 dist-test-slave-v0k1 krb5kdc[7031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:33 dist-test-slave-v0k1 krb5kdc[7031](info): set up 2 sockets
May 06 15:44:33 dist-test-slave-v0k1 krb5kdc[7031](info): commencing operation
krb5kdc: starting...
W20240506 15:44:33.959050 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:34 dist-test-slave-v0k1 krb5kdc[7031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:34.075381 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:34.155560 7044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:34.155783 7044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:34.155824 7044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:34.157294 7044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:34.157353 7044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:34.157377 7044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:34.157398 7044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:34.157418 7044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:34.159307 7044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:34.159575 7044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:34.159770 7044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:34.164757 7052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:34.165145 7050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:34.165902 7049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:34.166791 7044 server_base.cc:1034] running on GCE node
I20240506 15:44:34.167063 7044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:34.167317 7044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:34.168478 7044 hybrid_clock.cc:648] HybridClock initialized: now 1715010274168458 us; error 33 us; skew 500 ppm
May 06 15:44:34 dist-test-slave-v0k1 krb5kdc[7031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:34.170940 7044 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:34.171406 7044 webserver.cc:460] Webserver started at http://127.0.0.1:39045/ using document root <none> and password file <none>
I20240506 15:44:34.171624 7044 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:34.171672 7044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:34.171798 7044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:34.172844 7044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "ab9c6f43a38c447b8f32689e0683de56"
format_stamp: "Formatted at 2024-05-06 15:44:34 on dist-test-slave-v0k1"
server_key: "9bbd503709bef461647181e429fb97f3"
server_key_iv: "aa4a9e03b1d3ed564f736c3018f6bfb6"
server_key_version: "encryptionkey@0"
I20240506 15:44:34.173168 7044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "ab9c6f43a38c447b8f32689e0683de56"
format_stamp: "Formatted at 2024-05-06 15:44:34 on dist-test-slave-v0k1"
server_key: "9bbd503709bef461647181e429fb97f3"
server_key_iv: "aa4a9e03b1d3ed564f736c3018f6bfb6"
server_key_version: "encryptionkey@0"
I20240506 15:44:34.174412 7044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240506 15:44:34.175207 7059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:34.175381 7044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:34.175482 7044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "ab9c6f43a38c447b8f32689e0683de56"
format_stamp: "Formatted at 2024-05-06 15:44:34 on dist-test-slave-v0k1"
server_key: "9bbd503709bef461647181e429fb97f3"
server_key_iv: "aa4a9e03b1d3ed564f736c3018f6bfb6"
server_key_version: "encryptionkey@0"
I20240506 15:44:34.175546 7044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:34.179111 7044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:34.180994 7044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:34.181115 7044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:34.184860 7044 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39287
I20240506 15:44:34.184917 7111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39287 every 8 connection(s)
I20240506 15:44:34.185192 7044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:34.185981 7112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:34.188244 7112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56: Bootstrap starting.
I20240506 15:44:34.188802 7112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:34.189066 7112 log.cc:824] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:34.189543 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 7044
I20240506 15:44:34.189667 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:34.189853 7112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56: No bootstrap required, opened a new log
I20240506 15:44:34.189884 18371 external_mini_cluster.cc:1428] Setting key b1977a1d2394de4b4e5babce03d1bdd9
I20240506 15:44:34.192030 7112 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c6f43a38c447b8f32689e0683de56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } }
I20240506 15:44:34.192185 7112 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:34.192219 7112 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9c6f43a38c447b8f32689e0683de56, State: Initialized, Role: FOLLOWER
I20240506 15:44:34.192330 7112 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c6f43a38c447b8f32689e0683de56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } }
I20240506 15:44:34.192404 7112 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:34.192437 7112 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:34.192487 7112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:34.192598 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:34.190413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:35637 (local address 127.0.0.1:39287)
0506 15:44:34.190572 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:34.190581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:34.190601 (+ 20us) server_negotiation.cc:407] Connection header received
0506 15:44:34.191181 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:34.191188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:34.191217 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:34.191357 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:34.192095 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:34.192098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:34.192098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:34.192102 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:34.192123 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:34.192129 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:34.192202 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:34.192359 (+ 157us) server_negotiation.cc:299] Negotiation successful
0506 15:44:34.192410 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20240506 15:44:34.193162 7112 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c6f43a38c447b8f32689e0683de56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } }
I20240506 15:44:34.193269 7112 leader_election.cc:302] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab9c6f43a38c447b8f32689e0683de56; no voters:
I20240506 15:44:34.193434 7112 leader_election.cc:288] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:34.193487 7117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:34.193637 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:34.193666 7117 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [term 1 LEADER]: Becoming Leader. State: Replica: ab9c6f43a38c447b8f32689e0683de56, State: Running, Role: LEADER
I20240506 15:44:34.193698 7112 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:34.193754 7117 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c6f43a38c447b8f32689e0683de56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } }
I20240506 15:44:34.194070 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9c6f43a38c447b8f32689e0683de56. Latest consensus state: current_term: 1 leader_uuid: "ab9c6f43a38c447b8f32689e0683de56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c6f43a38c447b8f32689e0683de56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } } }
I20240506 15:44:34.194084 7118 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab9c6f43a38c447b8f32689e0683de56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c6f43a38c447b8f32689e0683de56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } } }
I20240506 15:44:34.194161 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:34.194185 7118 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56 [sys.catalog]: This master's current role is: LEADER
W20240506 15:44:34.195518 7137 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:34.195565 7137 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:34.195632 7122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:34.195809 7122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:34.197197 7122 catalog_manager.cc:1347] Generated new cluster ID: ea6fc97a6eaf4a1fbb26683ebd890dd7
I20240506 15:44:34.197249 7122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:34.200294 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:34.195239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:54723 (local address 127.0.0.1:39287)
0506 15:44:34.195273 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:34.195274 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:34.199202 (+ 3928us) server_negotiation.cc:407] Connection header received
0506 15:44:34.199256 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:34.199257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:34.199264 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:34.199337 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:34.200059 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:34.200061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:34.200061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:34.200066 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:34.200085 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:34.200090 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:34.200154 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:34.200208 (+ 54us) server_negotiation.cc:299] Negotiation successful
0506 15:44:34.200222 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20240506 15:44:34.201506 7122 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:34.201643 7122 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:34.211207 7122 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ab9c6f43a38c447b8f32689e0683de56: Generated new TSK 0
I20240506 15:44:34.211419 7122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:34.213526 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 7044
2024-05-06T15:44:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:34 dist-test-slave-v0k1 krb5kdc[7143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:34 dist-test-slave-v0k1 krb5kdc[7143](info): set up 2 sockets
May 06 15:44:34 dist-test-slave-v0k1 krb5kdc[7143](info): commencing operation
krb5kdc: starting...
W20240506 15:44:35.171375 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:35 dist-test-slave-v0k1 krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:35.292258 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:35.372079 7156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:35.372288 7156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:35.372311 7156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:35.373661 7156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:35.373713 7156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:35.373726 7156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:35.373739 7156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:35.373750 7156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:35.375449 7156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:35.375669 7156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:35.375877 7156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:35.380676 7162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:35.381120 7164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:35.381665 7161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:35.382601 7156 server_base.cc:1034] running on GCE node
I20240506 15:44:35.382848 7156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:35.383147 7156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:35.384325 7156 hybrid_clock.cc:648] HybridClock initialized: now 1715010275384270 us; error 48 us; skew 500 ppm
May 06 15:44:35 dist-test-slave-v0k1 krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:35.386837 7156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:35.387279 7156 webserver.cc:460] Webserver started at http://127.0.0.1:43395/ using document root <none> and password file <none>
I20240506 15:44:35.387460 7156 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:35.387497 7156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:35.387578 7156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:35.388585 7156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "ce173d3d13b34829822cdeaedd8c9dac"
format_stamp: "Formatted at 2024-05-06 15:44:35 on dist-test-slave-v0k1"
server_key: "eb7ef6c362e68a676ce3823d44986ae7"
server_key_iv: "f8b6c87bd486e9a2e2e58d6f30890792"
server_key_version: "encryptionkey@0"
I20240506 15:44:35.388862 7156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "ce173d3d13b34829822cdeaedd8c9dac"
format_stamp: "Formatted at 2024-05-06 15:44:35 on dist-test-slave-v0k1"
server_key: "eb7ef6c362e68a676ce3823d44986ae7"
server_key_iv: "f8b6c87bd486e9a2e2e58d6f30890792"
server_key_version: "encryptionkey@0"
I20240506 15:44:35.390056 7156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:44:35.390781 7171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:35.390949 7156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:35.391060 7156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "ce173d3d13b34829822cdeaedd8c9dac"
format_stamp: "Formatted at 2024-05-06 15:44:35 on dist-test-slave-v0k1"
server_key: "eb7ef6c362e68a676ce3823d44986ae7"
server_key_iv: "f8b6c87bd486e9a2e2e58d6f30890792"
server_key_version: "encryptionkey@0"
I20240506 15:44:35.391124 7156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:35.399233 7156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:35.401309 7156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:35.401440 7156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:35.405489 7156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33915
I20240506 15:44:35.405539 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33915 every 8 connection(s)
I20240506 15:44:35.405854 7156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:35.406790 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:35.406819 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 7156
I20240506 15:44:35.406929 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:35.407147 18371 external_mini_cluster.cc:1428] Setting key c154dce948cca04d46c9a8176eb240cd
I20240506 15:44:35.409987 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac: Bootstrap starting.
I20240506 15:44:35.410621 7224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:35.410954 7224 log.cc:824] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:35.411762 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac: No bootstrap required, opened a new log
May 06 15:44:35 dist-test-slave-v0k1 krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240506 15:44:35.413614 7224 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce173d3d13b34829822cdeaedd8c9dac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } }
I20240506 15:44:35.413762 7224 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:35.413791 7224 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce173d3d13b34829822cdeaedd8c9dac, State: Initialized, Role: FOLLOWER
I20240506 15:44:35.413897 7224 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce173d3d13b34829822cdeaedd8c9dac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } }
I20240506 15:44:35.414011 7224 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:35.414038 7224 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:35.414088 7224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:35.414829 7224 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce173d3d13b34829822cdeaedd8c9dac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } }
I20240506 15:44:35.414947 7224 leader_election.cc:302] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce173d3d13b34829822cdeaedd8c9dac; no voters:
I20240506 15:44:35.415143 7224 leader_election.cc:288] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:35.415196 7229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:35.415377 7224 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:35.415408 7229 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [term 1 LEADER]: Becoming Leader. State: Replica: ce173d3d13b34829822cdeaedd8c9dac, State: Running, Role: LEADER
I20240506 15:44:35.415565 7229 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce173d3d13b34829822cdeaedd8c9dac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } }
I20240506 15:44:35.415885 7230 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce173d3d13b34829822cdeaedd8c9dac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce173d3d13b34829822cdeaedd8c9dac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } } }
I20240506 15:44:35.415982 7230 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:35.416127 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce173d3d13b34829822cdeaedd8c9dac. Latest consensus state: current_term: 1 leader_uuid: "ce173d3d13b34829822cdeaedd8c9dac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce173d3d13b34829822cdeaedd8c9dac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } } }
I20240506 15:44:35.416179 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:35.416491 7236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:35.416949 7236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:35.417879 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:35.407663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:53773 (local address 127.0.0.1:33915)
0506 15:44:35.407806 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:35.407816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:35.407835 (+ 19us) server_negotiation.cc:407] Connection header received
0506 15:44:35.408292 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:35.408298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:35.408343 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:35.408457 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:35.409448 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:35.410159 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:35.411218 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:35.411521 (+ 303us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:35.413717 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:35.413725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:35.413725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:35.413744 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:35.415859 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:35.416379 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:35.416381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:35.416382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:35.416440 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:35.416714 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:35.416715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:35.416715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:35.416947 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:35.417592 (+ 645us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:35.417667 (+ 75us) server_negotiation.cc:299] Negotiation successful
0506 15:44:35.417744 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20240506 15:44:35.418671 7236 catalog_manager.cc:1347] Generated new cluster ID: a80be97f22134edc8c39ba2e108824b2
I20240506 15:44:35.418722 7236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:35.418972 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:35.428786 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:35.423402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:58577 (local address 127.0.0.1:33915)
0506 15:44:35.423442 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:35.423443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:35.423556 (+ 113us) server_negotiation.cc:407] Connection header received
0506 15:44:35.423614 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:35.423615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:35.423625 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:35.423672 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:35.424296 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:35.424762 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:35.425250 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:35.425441 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:35.426388 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:35.426393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:35.426394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:35.426400 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:35.427784 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:35.428121 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:35.428122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:35.428123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:35.428141 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:35.428315 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:35.428316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:35.428316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:35.428392 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:35.428482 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:35.428628 (+ 146us) server_negotiation.cc:299] Negotiation successful
0506 15:44:35.428693 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240506 15:44:35.429463 7236 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:35.430167 7236 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:35.435063 7236 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ce173d3d13b34829822cdeaedd8c9dac: Generated new TSK 0
I20240506 15:44:35.435235 7236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:35.443166 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 7156
W20240506 15:44:35.447583 7141 connection.cc:535] client connection to 127.0.0.1:33915 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33915): Connection reset by peer (error 104)
2024-05-06T15:44:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:35 dist-test-slave-v0k1 krb5kdc[7255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:35 dist-test-slave-v0k1 krb5kdc[7255](info): set up 2 sockets
May 06 15:44:35 dist-test-slave-v0k1 krb5kdc[7255](info): commencing operation
krb5kdc: starting...
W20240506 15:44:36.399312 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:36 dist-test-slave-v0k1 krb5kdc[7255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:36.516002 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:36.595335 7268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:36.595559 7268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:36.595597 7268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:36.596993 7268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:36.597045 7268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:36.597069 7268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:36.597088 7268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:36.597106 7268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:36.598822 7268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:36.599086 7268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:36.599288 7268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:36.603992 7273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:36.604705 7274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:36.605335 7276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:36.605904 7268 server_base.cc:1034] running on GCE node
I20240506 15:44:36.606166 7268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:36.606490 7268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:36.607657 7268 hybrid_clock.cc:648] HybridClock initialized: now 1715010276607640 us; error 37 us; skew 500 ppm
May 06 15:44:36 dist-test-slave-v0k1 krb5kdc[7255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:36.610121 7268 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:36.610535 7268 webserver.cc:460] Webserver started at http://127.0.0.1:42509/ using document root <none> and password file <none>
I20240506 15:44:36.610697 7268 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:36.610731 7268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:36.610819 7268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:36.611868 7268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "cf5f1fa21d5741d88dccd6971b753337"
format_stamp: "Formatted at 2024-05-06 15:44:36 on dist-test-slave-v0k1"
server_key: "7a64075dd2429c52ef0574fb5a5eb011"
server_key_iv: "aa75e7350dd604692798a0c3cf72e6b7"
server_key_version: "encryptionkey@0"
I20240506 15:44:36.612149 7268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "cf5f1fa21d5741d88dccd6971b753337"
format_stamp: "Formatted at 2024-05-06 15:44:36 on dist-test-slave-v0k1"
server_key: "7a64075dd2429c52ef0574fb5a5eb011"
server_key_iv: "aa75e7350dd604692798a0c3cf72e6b7"
server_key_version: "encryptionkey@0"
I20240506 15:44:36.613266 7268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:44:36.613922 7283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:36.614133 7268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:36.614212 7268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "cf5f1fa21d5741d88dccd6971b753337"
format_stamp: "Formatted at 2024-05-06 15:44:36 on dist-test-slave-v0k1"
server_key: "7a64075dd2429c52ef0574fb5a5eb011"
server_key_iv: "aa75e7350dd604692798a0c3cf72e6b7"
server_key_version: "encryptionkey@0"
I20240506 15:44:36.614264 7268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:36.636310 7268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:36.638074 7268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:36.638181 7268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:36.641820 7268 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40821
I20240506 15:44:36.641880 7335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40821 every 8 connection(s)
I20240506 15:44:36.642166 7268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:36.642998 7336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:36.645092 7336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337: Bootstrap starting.
I20240506 15:44:36.645643 7336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:36.645879 7336 log.cc:824] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:36.646512 7336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337: No bootstrap required, opened a new log
I20240506 15:44:36.647972 7336 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f1fa21d5741d88dccd6971b753337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20240506 15:44:36.648094 7336 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:36.648124 7336 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5f1fa21d5741d88dccd6971b753337, State: Initialized, Role: FOLLOWER
I20240506 15:44:36.648234 7336 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f1fa21d5741d88dccd6971b753337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20240506 15:44:36.648293 7336 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:36.648324 7336 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:36.648370 7336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:36.648896 7336 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f1fa21d5741d88dccd6971b753337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20240506 15:44:36.648988 7336 leader_election.cc:302] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5f1fa21d5741d88dccd6971b753337; no voters:
I20240506 15:44:36.649143 7336 leader_election.cc:288] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:36.649189 7339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:36.649360 7336 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:36.649617 7339 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [term 1 LEADER]: Becoming Leader. State: Replica: cf5f1fa21d5741d88dccd6971b753337, State: Running, Role: LEADER
I20240506 15:44:36.649705 7339 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f1fa21d5741d88dccd6971b753337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20240506 15:44:36.650054 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf5f1fa21d5741d88dccd6971b753337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f1fa21d5741d88dccd6971b753337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } } }
I20240506 15:44:36.650068 7341 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5f1fa21d5741d88dccd6971b753337. Latest consensus state: current_term: 1 leader_uuid: "cf5f1fa21d5741d88dccd6971b753337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f1fa21d5741d88dccd6971b753337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } } }
I20240506 15:44:36.650182 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:36.650187 7341 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:36.650381 7348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:36.650439 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 7268
I20240506 15:44:36.650544 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:36.650738 18371 external_mini_cluster.cc:1428] Setting key 504e2d77f868b678c52f5ed170749a3b
I20240506 15:44:36.651355 7348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:36.652909 7348 catalog_manager.cc:1347] Generated new cluster ID: 9a0871918df94175b7fee96522c52f63
I20240506 15:44:36.652980 7348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:36.663058 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:36.651216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:57531 (local address 127.0.0.1:40821)
0506 15:44:36.659134 (+ 7918us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:36.659144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:36.659168 (+ 24us) server_negotiation.cc:407] Connection header received
0506 15:44:36.659260 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:36.659265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:36.659304 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:36.659476 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:36.659852 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:36.660987 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:36.661609 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:36.661866 (+ 257us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:36.662372 (+ 506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:36.662379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:36.662379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:36.662588 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:36.662606 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:36.662613 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:36.662700 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:36.662813 (+ 113us) server_negotiation.cc:299] Negotiation successful
0506 15:44:36.662844 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7868,"thread_start_us":89,"threads_started":1}
I20240506 15:44:36.664064 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:36.664139 7348 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:36.664790 7348 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:36.670737 7348 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cf5f1fa21d5741d88dccd6971b753337: Generated new TSK 0
I20240506 15:44:36.670913 7348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:36.673255 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:36.665468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:53771 (local address 127.0.0.1:40821)
0506 15:44:36.665499 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:36.665500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:36.670463 (+ 4963us) server_negotiation.cc:407] Connection header received
0506 15:44:36.670538 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:36.670539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:36.670549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:36.670601 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:36.671304 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:36.671795 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:36.672317 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:36.672520 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:36.673002 (+ 482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:36.673004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:36.673005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:36.673010 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:36.673024 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:36.673029 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:36.673086 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:36.673171 (+ 85us) server_negotiation.cc:299] Negotiation successful
0506 15:44:36.673181 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240506 15:44:36.675125 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 7268
W20240506 15:44:36.679201 7253 connection.cc:535] client connection to 127.0.0.1:40821 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40821): Connection reset by peer (error 104)
2024-05-06T15:44:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1231 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:36 dist-test-slave-v0k1 krb5kdc[7367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:36 dist-test-slave-v0k1 krb5kdc[7367](info): set up 2 sockets
May 06 15:44:36 dist-test-slave-v0k1 krb5kdc[7367](info): commencing operation
krb5kdc: starting...
W20240506 15:44:37.631115 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:37 dist-test-slave-v0k1 krb5kdc[7367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240506 15:44:37.747507 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:37.826360 7380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:37.826575 7380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:37.826599 7380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:37.827965 7380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:37.828027 7380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:37.828040 7380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:37.828052 7380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:37.828063 7380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:37.829654 7380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:37.829834 7380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:37.829989 7380 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:37.835105 7386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:37.835389 7388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:37.836216 7385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:37.836833 7380 server_base.cc:1034] running on GCE node
I20240506 15:44:37.837069 7380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:37.837311 7380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:37.838444 7380 hybrid_clock.cc:648] HybridClock initialized: now 1715010277838426 us; error 33 us; skew 500 ppm
May 06 15:44:37 dist-test-slave-v0k1 krb5kdc[7367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:37.840835 7380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240506 15:44:37.841214 7380 webserver.cc:460] Webserver started at http://127.0.0.1:37161/ using document root <none> and password file <none>
I20240506 15:44:37.841406 7380 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:37.841447 7380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:37.841532 7380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:37.842481 7380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "feef995d5cd043b9b51d7d7ce3425773"
format_stamp: "Formatted at 2024-05-06 15:44:37 on dist-test-slave-v0k1"
server_key: "4459f5c74441c6701199d29e1f8ff525"
server_key_iv: "65a9658b972856a1b37fa7950da9cb6a"
server_key_version: "encryptionkey@0"
I20240506 15:44:37.842746 7380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "feef995d5cd043b9b51d7d7ce3425773"
format_stamp: "Formatted at 2024-05-06 15:44:37 on dist-test-slave-v0k1"
server_key: "4459f5c74441c6701199d29e1f8ff525"
server_key_iv: "65a9658b972856a1b37fa7950da9cb6a"
server_key_version: "encryptionkey@0"
I20240506 15:44:37.843928 7380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:44:37.844566 7395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:37.844736 7380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:37.844812 7380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "feef995d5cd043b9b51d7d7ce3425773"
format_stamp: "Formatted at 2024-05-06 15:44:37 on dist-test-slave-v0k1"
server_key: "4459f5c74441c6701199d29e1f8ff525"
server_key_iv: "65a9658b972856a1b37fa7950da9cb6a"
server_key_version: "encryptionkey@0"
I20240506 15:44:37.844868 7380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:37.848338 7380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:37.850440 7380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:37.850548 7380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:37.854207 7380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38577
I20240506 15:44:37.854252 7447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38577 every 8 connection(s)
I20240506 15:44:37.854544 7380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:37.855440 7448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:37.857546 7448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773: Bootstrap starting.
I20240506 15:44:37.858124 7448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:37.858366 7448 log.cc:824] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:37.859051 7448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773: No bootstrap required, opened a new log
I20240506 15:44:37.860502 7448 raft_consensus.cc:348] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feef995d5cd043b9b51d7d7ce3425773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38577 } }
I20240506 15:44:37.860631 7448 raft_consensus.cc:374] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:37.860648 7448 raft_consensus.cc:729] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feef995d5cd043b9b51d7d7ce3425773, State: Initialized, Role: FOLLOWER
I20240506 15:44:37.860781 7448 consensus_queue.cc:260] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feef995d5cd043b9b51d7d7ce3425773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38577 } }
I20240506 15:44:37.860843 7448 raft_consensus.cc:388] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:37.860877 7448 raft_consensus.cc:482] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:37.860924 7448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:37.861512 7448 raft_consensus.cc:504] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feef995d5cd043b9b51d7d7ce3425773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38577 } }
I20240506 15:44:37.861618 7448 leader_election.cc:302] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feef995d5cd043b9b51d7d7ce3425773; no voters:
I20240506 15:44:37.861765 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 7380
I20240506 15:44:37.861804 7448 leader_election.cc:288] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:37.861860 7451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:37.861898 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:37.862025 7451 raft_consensus.cc:686] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [term 1 LEADER]: Becoming Leader. State: Replica: feef995d5cd043b9b51d7d7ce3425773, State: Running, Role: LEADER
I20240506 15:44:37.862075 18371 external_mini_cluster.cc:1428] Setting key 6e73dfed6e6bec5a3bb3f8b435a5df0f
I20240506 15:44:37.862071 7448 sys_catalog.cc:564] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:37.862147 7451 consensus_queue.cc:237] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feef995d5cd043b9b51d7d7ce3425773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38577 } }
I20240506 15:44:37.862404 7451 sys_catalog.cc:455] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feef995d5cd043b9b51d7d7ce3425773. Latest consensus state: current_term: 1 leader_uuid: "feef995d5cd043b9b51d7d7ce3425773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feef995d5cd043b9b51d7d7ce3425773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38577 } } }
I20240506 15:44:37.862491 7451 sys_catalog.cc:458] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:37.862538 7452 sys_catalog.cc:455] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feef995d5cd043b9b51d7d7ce3425773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feef995d5cd043b9b51d7d7ce3425773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38577 } } }
I20240506 15:44:37.862596 7452 sys_catalog.cc:458] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:37.863211 7457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:37.863811 7457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:37.865054 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:37.862719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:40465 (local address 127.0.0.1:38577)
0506 15:44:37.862868 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:37.862883 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:37.862975 (+ 92us) server_negotiation.cc:407] Connection header received
0506 15:44:37.863193 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:37.863200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:37.863233 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:37.863383 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:37.864513 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:37.864516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:37.864517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:37.864521 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:37.864551 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:37.864559 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:37.864621 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:37.864845 (+ 224us) server_negotiation.cc:299] Negotiation successful
0506 15:44:37.864885 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":80,"threads_started":1}
I20240506 15:44:37.865890 7457 catalog_manager.cc:1347] Generated new cluster ID: d208a4a239734f82b8b533329b3933b9
I20240506 15:44:37.865943 7457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:37.867189 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:37.869953 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:37.868809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:49037 (local address 127.0.0.1:38577)
0506 15:44:37.868845 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:37.868847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:37.868899 (+ 52us) server_negotiation.cc:407] Connection header received
0506 15:44:37.868991 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:37.868992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:37.868997 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:37.869051 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:37.869735 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:37.869736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:37.869736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:37.869741 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:37.869757 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:37.869762 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:37.869822 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:37.869877 (+ 55us) server_negotiation.cc:299] Negotiation successful
0506 15:44:37.869890 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:44:37.870839 7457 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:37.870982 7457 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:37.875731 7457 catalog_manager.cc:5857] T 00000000000000000000000000000000 P feef995d5cd043b9b51d7d7ce3425773: Generated new TSK 0
I20240506 15:44:37.875886 7457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:37.880702 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 7380
2024-05-06T15:44:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:37 dist-test-slave-v0k1 krb5kdc[7479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:37 dist-test-slave-v0k1 krb5kdc[7479](info): set up 2 sockets
May 06 15:44:37 dist-test-slave-v0k1 krb5kdc[7479](info): commencing operation
krb5kdc: starting...
W20240506 15:44:38.831395 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:38 dist-test-slave-v0k1 krb5kdc[7479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:38.947324 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43151
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:43151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:39.025507 7492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:39.025687 7492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:39.025720 7492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:39.026989 7492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:39.027046 7492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:39.027069 7492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:39.027088 7492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:39.027105 7492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:39.028671 7492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:43151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:43151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:39.028892 7492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:39.029078 7492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:39.033612 7498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:39.034071 7500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:39.034617 7497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:39.035393 7492 server_base.cc:1034] running on GCE node
I20240506 15:44:39.035624 7492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:39.035851 7492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:39.036978 7492 hybrid_clock.cc:648] HybridClock initialized: now 1715010279036960 us; error 32 us; skew 500 ppm
May 06 15:44:39 dist-test-slave-v0k1 krb5kdc[7479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010279, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:39.049701 7492 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:39.050130 7492 webserver.cc:460] Webserver started at http://127.17.240.254:45949/ using document root <none> and password file <none>
I20240506 15:44:39.050312 7492 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:39.050359 7492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:39.050480 7492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:39.051594 7492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "7c63572f4c054c749f54fe4351c875c1"
format_stamp: "Formatted at 2024-05-06 15:44:39 on dist-test-slave-v0k1"
server_key: "411e96919422ecba3ecb26c81f075b4d"
server_key_iv: "826b799a03c10028f3da474f754511e4"
server_key_version: "encryptionkey@0"
I20240506 15:44:39.051892 7492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "7c63572f4c054c749f54fe4351c875c1"
format_stamp: "Formatted at 2024-05-06 15:44:39 on dist-test-slave-v0k1"
server_key: "411e96919422ecba3ecb26c81f075b4d"
server_key_iv: "826b799a03c10028f3da474f754511e4"
server_key_version: "encryptionkey@0"
I20240506 15:44:39.053094 7492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:44:39.053777 7507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:39.053978 7492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:39.054055 7492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "7c63572f4c054c749f54fe4351c875c1"
format_stamp: "Formatted at 2024-05-06 15:44:39 on dist-test-slave-v0k1"
server_key: "411e96919422ecba3ecb26c81f075b4d"
server_key_iv: "826b799a03c10028f3da474f754511e4"
server_key_version: "encryptionkey@0"
I20240506 15:44:39.054116 7492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:39.076570 7492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:39.078322 7492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:39.078449 7492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:39.082244 7492 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:43151
I20240506 15:44:39.082278 7559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:43151 every 8 connection(s)
I20240506 15:44:39.082599 7492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:39.083446 7560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:39.085512 7560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1: Bootstrap starting.
I20240506 15:44:39.086023 7560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:39.086233 7560 log.cc:824] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:39.086840 7560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1: No bootstrap required, opened a new log
I20240506 15:44:39.088219 7560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c63572f4c054c749f54fe4351c875c1" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43151 } }
I20240506 15:44:39.088338 7560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:39.088356 7560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c63572f4c054c749f54fe4351c875c1, State: Initialized, Role: FOLLOWER
I20240506 15:44:39.088446 7560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c63572f4c054c749f54fe4351c875c1" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43151 } }
I20240506 15:44:39.088497 7560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:39.088517 7560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:39.088541 7560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:39.089030 7560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c63572f4c054c749f54fe4351c875c1" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43151 } }
I20240506 15:44:39.089130 7560 leader_election.cc:302] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c63572f4c054c749f54fe4351c875c1; no voters:
I20240506 15:44:39.089257 7560 leader_election.cc:288] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:39.089316 7563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:39.089468 7560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:39.089787 7563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [term 1 LEADER]: Becoming Leader. State: Replica: 7c63572f4c054c749f54fe4351c875c1, State: Running, Role: LEADER
I20240506 15:44:39.089901 7563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c63572f4c054c749f54fe4351c875c1" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43151 } }
I20240506 15:44:39.090307 7564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c63572f4c054c749f54fe4351c875c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c63572f4c054c749f54fe4351c875c1" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43151 } } }
I20240506 15:44:39.090337 7565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c63572f4c054c749f54fe4351c875c1. Latest consensus state: current_term: 1 leader_uuid: "7c63572f4c054c749f54fe4351c875c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c63572f4c054c749f54fe4351c875c1" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 43151 } } }
I20240506 15:44:39.090397 7564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:39.090409 7565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:39.090618 7572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:39.091152 7572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:39.092097 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 7492
I20240506 15:44:39.092196 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:39.092356 18371 external_mini_cluster.cc:1428] Setting key 6b34bcbbbe08c69014e10ce2352d7167
I20240506 15:44:39.092557 7572 catalog_manager.cc:1347] Generated new cluster ID: 4422d37fbcba46498e3955ca02c92b81
I20240506 15:44:39.092608 7572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:39.097548 7572 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:39.098019 7572 catalog_manager.cc:1504] Loading token signing keys...
May 06 15:44:39 dist-test-slave-v0k1 krb5kdc[7479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:39.102962 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:39.093003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53021 (local address 127.17.240.254:43151)
0506 15:44:39.093225 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:39.093235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:39.093257 (+ 22us) server_negotiation.cc:407] Connection header received
0506 15:44:39.093352 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:39.093357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:39.093400 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:39.093531 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:39.094477 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:39.095637 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:39.096427 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:39.097063 (+ 636us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:39.099327 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:39.099332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:39.099333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:39.099351 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:39.101342 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:39.101736 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:39.101739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:39.101739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:39.101811 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:39.101976 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:39.101977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:39.101977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:39.102162 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:39.102276 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:39.102671 (+ 395us) server_negotiation.cc:299] Negotiation successful
0506 15:44:39.102767 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20240506 15:44:39.103962 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:39.106637 7572 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7c63572f4c054c749f54fe4351c875c1: Generated new TSK 0
I20240506 15:44:39.106791 7572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:39.111449 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:39.105305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57133 (local address 127.17.240.254:43151)
0506 15:44:39.105339 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:39.105340 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:39.105448 (+ 108us) server_negotiation.cc:407] Connection header received
0506 15:44:39.105526 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:39.105526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:39.105535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:39.105593 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:39.106380 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:39.107135 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:39.107880 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:39.107986 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:39.109128 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:39.109132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:39.109132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:39.109137 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:39.110451 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:39.110817 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:39.110818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:39.110818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:39.110834 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:39.110996 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:39.110997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:39.110997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:39.111103 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:39.111189 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:39.111344 (+ 155us) server_negotiation.cc:299] Negotiation successful
0506 15:44:39.111390 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240506 15:44:39.113507 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 7492
W20240506 15:44:39.118113 7477 connection.cc:535] client connection to 127.17.240.254:43151 recv error: Network error: failed to read from TLS socket (remote: 127.17.240.254:43151): Connection reset by peer (error 104)
2024-05-06T15:44:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:39 dist-test-slave-v0k1 krb5kdc[7591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:39 dist-test-slave-v0k1 krb5kdc[7591](info): set up 2 sockets
May 06 15:44:39 dist-test-slave-v0k1 krb5kdc[7591](info): commencing operation
krb5kdc: starting...
W20240506 15:44:40.067333 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:40 dist-test-slave-v0k1 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:40.185999 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39781
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:39781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:40.264555 7604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:40.264752 7604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:40.264776 7604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:40.266064 7604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:40.266101 7604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:40.266114 7604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:40.266126 7604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:40.266144 7604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:40.267719 7604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:39781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:39781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:40.267903 7604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:40.268061 7604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:40.272486 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:40.273015 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:40.273923 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:40.274688 7604 server_base.cc:1034] running on GCE node
I20240506 15:44:40.274945 7604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:40.275210 7604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:40.276369 7604 hybrid_clock.cc:648] HybridClock initialized: now 1715010280276322 us; error 39 us; skew 500 ppm
May 06 15:44:40 dist-test-slave-v0k1 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010280, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:40.278720 7604 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:40.279129 7604 webserver.cc:460] Webserver started at http://127.17.240.254:35753/ using document root <none> and password file <none>
I20240506 15:44:40.279302 7604 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:40.279338 7604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:40.279428 7604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:40.280428 7604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "b93670af8121443a9ed0715e94343bd0"
format_stamp: "Formatted at 2024-05-06 15:44:40 on dist-test-slave-v0k1"
server_key: "7a2e800c9370512a52f54cb646e32fb5"
server_key_iv: "077b6ae8fe6611246a4a2b9e053d6f7d"
server_key_version: "encryptionkey@0"
I20240506 15:44:40.280694 7604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "b93670af8121443a9ed0715e94343bd0"
format_stamp: "Formatted at 2024-05-06 15:44:40 on dist-test-slave-v0k1"
server_key: "7a2e800c9370512a52f54cb646e32fb5"
server_key_iv: "077b6ae8fe6611246a4a2b9e053d6f7d"
server_key_version: "encryptionkey@0"
I20240506 15:44:40.281849 7604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:44:40.282555 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:40.282732 7604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:40.282804 7604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "b93670af8121443a9ed0715e94343bd0"
format_stamp: "Formatted at 2024-05-06 15:44:40 on dist-test-slave-v0k1"
server_key: "7a2e800c9370512a52f54cb646e32fb5"
server_key_iv: "077b6ae8fe6611246a4a2b9e053d6f7d"
server_key_version: "encryptionkey@0"
I20240506 15:44:40.282857 7604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:40.301776 7604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:40.304016 7604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:40.304121 7604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:40.307873 7604 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:39781
I20240506 15:44:40.307942 7671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:39781 every 8 connection(s)
I20240506 15:44:40.308238 7604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:40.309074 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:40.310288 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 7604
I20240506 15:44:40.310380 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:40.310552 18371 external_mini_cluster.cc:1428] Setting key 5004aa26b95a7b0078df669c6cc9059f
I20240506 15:44:40.311655 7672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0: Bootstrap starting.
I20240506 15:44:40.312333 7672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:40.312587 7672 log.cc:824] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:40.313203 7672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0: No bootstrap required, opened a new log
I20240506 15:44:40.314965 7672 raft_consensus.cc:348] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93670af8121443a9ed0715e94343bd0" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39781 } }
I20240506 15:44:40.315157 7672 raft_consensus.cc:374] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:40.315177 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:40.311091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55111 (local address 127.17.240.254:39781)
0506 15:44:40.311305 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:40.311317 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:40.311344 (+ 27us) server_negotiation.cc:407] Connection header received
0506 15:44:40.311892 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:40.311902 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:40.311968 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:40.312134 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:40.312483 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:40.313225 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:40.313802 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:40.314056 (+ 254us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:40.314540 (+ 484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:40.314549 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:40.314549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:40.314703 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:40.314724 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:40.314731 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:40.314817 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:40.314934 (+ 117us) server_negotiation.cc:299] Negotiation successful
0506 15:44:40.314961 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20240506 15:44:40.315191 7672 raft_consensus.cc:729] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93670af8121443a9ed0715e94343bd0, State: Initialized, Role: FOLLOWER
I20240506 15:44:40.315318 7672 consensus_queue.cc:260] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93670af8121443a9ed0715e94343bd0" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39781 } }
I20240506 15:44:40.315387 7672 raft_consensus.cc:388] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:40.315421 7672 raft_consensus.cc:482] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:40.315500 7672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:40.316102 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:40.316152 7672 raft_consensus.cc:504] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93670af8121443a9ed0715e94343bd0" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39781 } }
I20240506 15:44:40.316262 7672 leader_election.cc:302] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b93670af8121443a9ed0715e94343bd0; no voters:
I20240506 15:44:40.316423 7672 leader_election.cc:288] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:40.316478 7677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:40.316612 7677 raft_consensus.cc:686] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [term 1 LEADER]: Becoming Leader. State: Replica: b93670af8121443a9ed0715e94343bd0, State: Running, Role: LEADER
I20240506 15:44:40.316651 7672 sys_catalog.cc:564] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:40.316777 7677 consensus_queue.cc:237] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93670af8121443a9ed0715e94343bd0" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39781 } }
I20240506 15:44:40.317075 7679 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b93670af8121443a9ed0715e94343bd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93670af8121443a9ed0715e94343bd0" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39781 } } }
I20240506 15:44:40.317167 7679 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:40.318531 7678 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b93670af8121443a9ed0715e94343bd0. Latest consensus state: current_term: 1 leader_uuid: "b93670af8121443a9ed0715e94343bd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93670af8121443a9ed0715e94343bd0" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 39781 } } }
W20240506 15:44:40.318578 7698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:40.318732 7698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:40.318676 7678 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:40.318869 7687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:40.319100 7687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:40.320600 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:40.317686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34173 (local address 127.17.240.254:39781)
0506 15:44:40.317723 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:40.317725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:40.317975 (+ 250us) server_negotiation.cc:407] Connection header received
0506 15:44:40.318041 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:40.318041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:40.318049 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:40.318098 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:40.318480 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:40.319155 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:40.319665 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:40.319842 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:40.320321 (+ 479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:40.320323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:40.320324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:40.320328 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:40.320342 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:40.320346 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:40.320423 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:40.320516 (+ 93us) server_negotiation.cc:299] Negotiation successful
0506 15:44:40.320527 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:44:40.320641 7687 catalog_manager.cc:1347] Generated new cluster ID: 4795aedaf1244533a6b40864433fc1d5
I20240506 15:44:40.320684 7687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:40.333070 7687 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:40.333712 7687 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:40.338369 7687 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b93670af8121443a9ed0715e94343bd0: Generated new TSK 0
I20240506 15:44:40.338550 7687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:40.344136 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 7604
W20240506 15:44:40.348369 7589 connection.cc:535] client connection to 127.17.240.254:39781 recv error: Network error: failed to read from TLS socket (remote: 127.17.240.254:39781): Connection reset by peer (error 104)
2024-05-06T15:44:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:40 dist-test-slave-v0k1 krb5kdc[7703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:40 dist-test-slave-v0k1 krb5kdc[7703](info): set up 2 sockets
May 06 15:44:40 dist-test-slave-v0k1 krb5kdc[7703](info): commencing operation
krb5kdc: starting...
W20240506 15:44:41.303292 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:41 dist-test-slave-v0k1 krb5kdc[7703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:41.422287 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36101
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:41.512706 7716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:41.512912 7716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:41.512949 7716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:41.514205 7716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:41.514248 7716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:41.514271 7716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:41.514290 7716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:41.514308 7716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:41.515918 7716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:36101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:41.516141 7716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:41.516332 7716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:41.520884 7722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:41.521550 7724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:41.522094 7721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:41.522738 7716 server_base.cc:1034] running on GCE node
I20240506 15:44:41.522961 7716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:41.523219 7716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:41.524350 7716 hybrid_clock.cc:648] HybridClock initialized: now 1715010281524323 us; error 44 us; skew 500 ppm
May 06 15:44:41 dist-test-slave-v0k1 krb5kdc[7703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010281, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:41.526823 7716 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:41.527274 7716 webserver.cc:460] Webserver started at http://127.17.240.254:43833/ using document root <none> and password file <none>
I20240506 15:44:41.527482 7716 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:41.527526 7716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:41.527639 7716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:41.528627 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "3fe8b64167cd4848987aecdb62f41dc9"
format_stamp: "Formatted at 2024-05-06 15:44:41 on dist-test-slave-v0k1"
server_key: "84f0936245ea20d379b80b96094d6923"
server_key_iv: "677fd3268a16621669e6987a8038c19a"
server_key_version: "encryptionkey@0"
I20240506 15:44:41.528929 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "3fe8b64167cd4848987aecdb62f41dc9"
format_stamp: "Formatted at 2024-05-06 15:44:41 on dist-test-slave-v0k1"
server_key: "84f0936245ea20d379b80b96094d6923"
server_key_iv: "677fd3268a16621669e6987a8038c19a"
server_key_version: "encryptionkey@0"
I20240506 15:44:41.530153 7716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240506 15:44:41.530874 7731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:41.531064 7716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:41.531139 7716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "3fe8b64167cd4848987aecdb62f41dc9"
format_stamp: "Formatted at 2024-05-06 15:44:41 on dist-test-slave-v0k1"
server_key: "84f0936245ea20d379b80b96094d6923"
server_key_iv: "677fd3268a16621669e6987a8038c19a"
server_key_version: "encryptionkey@0"
I20240506 15:44:41.531206 7716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:41.534703 7716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:41.536942 7716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:41.537060 7716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:41.541263 7716 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:36101
I20240506 15:44:41.541301 7783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:36101 every 8 connection(s)
I20240506 15:44:41.541606 7716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:41.542444 7784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:41.544565 7784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9: Bootstrap starting.
I20240506 15:44:41.545125 7784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:41.545368 7784 log.cc:824] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:41.546010 7784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9: No bootstrap required, opened a new log
I20240506 15:44:41.547608 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 7716
I20240506 15:44:41.547523 7784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe8b64167cd4848987aecdb62f41dc9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36101 } }
I20240506 15:44:41.547669 7784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:41.547699 7784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe8b64167cd4848987aecdb62f41dc9, State: Initialized, Role: FOLLOWER
I20240506 15:44:41.547730 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:41.547847 7784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe8b64167cd4848987aecdb62f41dc9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36101 } }
I20240506 15:44:41.547905 18371 external_mini_cluster.cc:1428] Setting key aedab9486fc00af9539221bc23674309
I20240506 15:44:41.547923 7784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:41.547963 7784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:41.548019 7784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:41.548636 7784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe8b64167cd4848987aecdb62f41dc9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36101 } }
I20240506 15:44:41.548739 7784 leader_election.cc:302] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fe8b64167cd4848987aecdb62f41dc9; no voters:
I20240506 15:44:41.548882 7784 leader_election.cc:288] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:41.548952 7789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:41.549125 7784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:41.549185 7789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [term 1 LEADER]: Becoming Leader. State: Replica: 3fe8b64167cd4848987aecdb62f41dc9, State: Running, Role: LEADER
I20240506 15:44:41.549347 7789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe8b64167cd4848987aecdb62f41dc9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36101 } }
I20240506 15:44:41.549702 7789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fe8b64167cd4848987aecdb62f41dc9. Latest consensus state: current_term: 1 leader_uuid: "3fe8b64167cd4848987aecdb62f41dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe8b64167cd4848987aecdb62f41dc9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36101 } } }
I20240506 15:44:41.549813 7789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:41.549717 7790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fe8b64167cd4848987aecdb62f41dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe8b64167cd4848987aecdb62f41dc9" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 36101 } } }
I20240506 15:44:41.549968 7790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:41.550055 7798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:41.550559 7798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:41.551795 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:41.548369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40655 (local address 127.17.240.254:36101)
0506 15:44:41.548626 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:41.548636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:41.548658 (+ 22us) server_negotiation.cc:407] Connection header received
0506 15:44:41.548772 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:41.548778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:41.548816 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:41.548978 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:41.551423 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:41.551426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:41.551426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:41.551431 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:41.551455 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:41.551462 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:41.551531 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:41.551622 (+ 91us) server_negotiation.cc:299] Negotiation successful
0506 15:44:41.551658 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":144,"threads_started":1}
I20240506 15:44:41.552277 7798 catalog_manager.cc:1347] Generated new cluster ID: 447092081eeb4986ba2cc2265b8c60d4
I20240506 15:44:41.552332 7798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:41.552611 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:41.563524 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:41.562160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57075 (local address 127.17.240.254:36101)
0506 15:44:41.562213 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:41.562215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:41.562316 (+ 101us) server_negotiation.cc:407] Connection header received
0506 15:44:41.562388 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:41.562389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:41.562398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:41.562452 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:41.563229 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:41.563230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:41.563230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:41.563236 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:41.563270 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:41.563275 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:41.563310 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:41.563429 (+ 119us) server_negotiation.cc:299] Negotiation successful
0506 15:44:41.563444 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240506 15:44:41.564664 7798 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:41.564811 7798 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:41.570065 7798 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3fe8b64167cd4848987aecdb62f41dc9: Generated new TSK 0
I20240506 15:44:41.570238 7798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:41.573376 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 7716
2024-05-06T15:44:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:41 dist-test-slave-v0k1 krb5kdc[7815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:41 dist-test-slave-v0k1 krb5kdc[7815](info): set up 2 sockets
May 06 15:44:41 dist-test-slave-v0k1 krb5kdc[7815](info): commencing operation
krb5kdc: starting...
W20240506 15:44:42.519332 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:42 dist-test-slave-v0k1 krb5kdc[7815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:42.634106 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:46881
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:46881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:42.713837 7828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:42.714041 7828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:42.714064 7828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:42.715354 7828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:42.715399 7828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:42.715411 7828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:42.715423 7828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:42.715435 7828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:42.717025 7828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:46881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:46881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:42.717212 7828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:42.717376 7828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:42.722028 7833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:42.722360 7836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:42.722844 7834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:42.723723 7828 server_base.cc:1034] running on GCE node
I20240506 15:44:42.723960 7828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:42.724195 7828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:42.725327 7828 hybrid_clock.cc:648] HybridClock initialized: now 1715010282725310 us; error 35 us; skew 500 ppm
May 06 15:44:42 dist-test-slave-v0k1 krb5kdc[7815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010282, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:42.727761 7828 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:42.728137 7828 webserver.cc:460] Webserver started at http://127.17.240.254:39681/ using document root <none> and password file <none>
I20240506 15:44:42.728317 7828 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:42.728358 7828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:42.728447 7828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:42.729369 7828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "65ea6240d98f4158b19d11aa37fd7c2d"
format_stamp: "Formatted at 2024-05-06 15:44:42 on dist-test-slave-v0k1"
server_key: "fd1c3668a435126874cb7667d569419b"
server_key_iv: "a2e689d2d84fc1fc2a1de5436b7aa2dd"
server_key_version: "encryptionkey@0"
I20240506 15:44:42.729689 7828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "65ea6240d98f4158b19d11aa37fd7c2d"
format_stamp: "Formatted at 2024-05-06 15:44:42 on dist-test-slave-v0k1"
server_key: "fd1c3668a435126874cb7667d569419b"
server_key_iv: "a2e689d2d84fc1fc2a1de5436b7aa2dd"
server_key_version: "encryptionkey@0"
I20240506 15:44:42.730834 7828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:44:42.731559 7843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:42.731734 7828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:42.731812 7828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "65ea6240d98f4158b19d11aa37fd7c2d"
format_stamp: "Formatted at 2024-05-06 15:44:42 on dist-test-slave-v0k1"
server_key: "fd1c3668a435126874cb7667d569419b"
server_key_iv: "a2e689d2d84fc1fc2a1de5436b7aa2dd"
server_key_version: "encryptionkey@0"
I20240506 15:44:42.731868 7828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:42.746923 7828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:42.749100 7828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:42.749212 7828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:42.752679 7828 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:46881
I20240506 15:44:42.752737 7895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:46881 every 8 connection(s)
I20240506 15:44:42.753034 7828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:42.753849 7896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:42.756042 7896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d: Bootstrap starting.
I20240506 15:44:42.756596 7896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:42.756843 7896 log.cc:824] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:42.757547 7896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d: No bootstrap required, opened a new log
I20240506 15:44:42.758708 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 7828
I20240506 15:44:42.758792 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:42.758949 18371 external_mini_cluster.cc:1428] Setting key d7361c428e1f38425ee15c4dff436bb1
I20240506 15:44:42.759127 7896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea6240d98f4158b19d11aa37fd7c2d" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46881 } }
I20240506 15:44:42.759254 7896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:42.759282 7896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ea6240d98f4158b19d11aa37fd7c2d, State: Initialized, Role: FOLLOWER
I20240506 15:44:42.759404 7896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea6240d98f4158b19d11aa37fd7c2d" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46881 } }
I20240506 15:44:42.759487 7896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:42.759550 7896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:42.759609 7896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:42.760327 7896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea6240d98f4158b19d11aa37fd7c2d" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46881 } }
I20240506 15:44:42.760418 7896 leader_election.cc:302] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ea6240d98f4158b19d11aa37fd7c2d; no voters:
I20240506 15:44:42.760560 7896 leader_election.cc:288] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:42.760757 7901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:42.760828 7896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:42.760944 7901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [term 1 LEADER]: Becoming Leader. State: Replica: 65ea6240d98f4158b19d11aa37fd7c2d, State: Running, Role: LEADER
I20240506 15:44:42.761071 7901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea6240d98f4158b19d11aa37fd7c2d" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46881 } }
I20240506 15:44:42.761417 7902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65ea6240d98f4158b19d11aa37fd7c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea6240d98f4158b19d11aa37fd7c2d" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46881 } } }
I20240506 15:44:42.761656 7902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:42.761410 7901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ea6240d98f4158b19d11aa37fd7c2d. Latest consensus state: current_term: 1 leader_uuid: "65ea6240d98f4158b19d11aa37fd7c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea6240d98f4158b19d11aa37fd7c2d" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46881 } } }
I20240506 15:44:42.761924 7901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d [sys.catalog]: This master's current role is: LEADER
W20240506 15:44:42.762712 7917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:42.762820 7917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:42.762918 7910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:42.763490 7910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
May 06 15:44:42 dist-test-slave-v0k1 krb5kdc[7815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:42.765138 7910 catalog_manager.cc:1347] Generated new cluster ID: 222dcebe7b3b4a2aa2ec3ffdbabdbef2
I20240506 15:44:42.765197 7910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:42.768539 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:42.759468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38951 (local address 127.17.240.254:46881)
0506 15:44:42.759675 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:42.759687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:42.759711 (+ 24us) server_negotiation.cc:407] Connection header received
0506 15:44:42.759812 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:42.759818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:42.759868 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:42.760024 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:42.760811 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:42.761964 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:42.762814 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:42.763072 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:42.765500 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:42.765521 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:42.765521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:42.765538 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:42.767275 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:42.767595 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:42.767597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:42.767598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:42.767642 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:42.767782 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:42.767783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:42.767783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:42.767960 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:42.768093 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:42.768209 (+ 116us) server_negotiation.cc:299] Negotiation successful
0506 15:44:42.768338 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":110,"threads_started":1}
I20240506 15:44:42.769487 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:42.775962 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:42.770717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38963 (local address 127.17.240.254:46881)
0506 15:44:42.770768 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:42.770769 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:42.770877 (+ 108us) server_negotiation.cc:407] Connection header received
0506 15:44:42.770934 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:42.770935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:42.770949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:42.771001 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:42.771633 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:42.772088 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:42.772586 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:42.773086 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:42.773648 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:42.773650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:42.773651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:42.773656 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:42.774972 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:42.775317 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:42.775318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:42.775318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:42.775342 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:42.775480 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:42.775481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:42.775481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:42.775569 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:42.775668 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:42.775768 (+ 100us) server_negotiation.cc:299] Negotiation successful
0506 15:44:42.775847 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:44:42.776988 7910 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:42.777477 7910 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:42.780929 7910 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 65ea6240d98f4158b19d11aa37fd7c2d: Generated new TSK 0
I20240506 15:44:42.781095 7910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:42.783923 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 7828
2024-05-06T15:44:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:42 dist-test-slave-v0k1 krb5kdc[7927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:42 dist-test-slave-v0k1 krb5kdc[7927](info): set up 2 sockets
May 06 15:44:42 dist-test-slave-v0k1 krb5kdc[7927](info): commencing operation
krb5kdc: starting...
W20240506 15:44:43.739391 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:43 dist-test-slave-v0k1 krb5kdc[7927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:43.867393 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:42067
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:39939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:43.948480 7940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:43.948683 7940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:43.948706 7940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:43.950057 7940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:43.950107 7940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:43.950121 7940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:43.950134 7940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:43.950145 7940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:43.951851 7940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:39939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:42067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:43.952070 7940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:43.952271 7940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:43.956946 7945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:43.957568 7946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:43.958153 7948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:43.958775 7940 server_base.cc:1034] running on GCE node
I20240506 15:44:43.959074 7940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:43.959342 7940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:43.960564 7940 hybrid_clock.cc:648] HybridClock initialized: now 1715010283960544 us; error 36 us; skew 500 ppm
May 06 15:44:43 dist-test-slave-v0k1 krb5kdc[7927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010283, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:43.963191 7940 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:43.963579 7940 webserver.cc:460] Webserver started at http://127.17.240.254:36243/ using document root <none> and password file <none>
I20240506 15:44:43.963737 7940 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:43.963773 7940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:43.963856 7940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:43.964810 7940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "62fe781507f347b38cb96307937f891e"
format_stamp: "Formatted at 2024-05-06 15:44:43 on dist-test-slave-v0k1"
server_key: "08ccb1853ef4fde42a749d06f5944a1d"
server_key_iv: "435510bbf03858d9233feaded8b9df9a"
server_key_version: "encryptionkey@0"
I20240506 15:44:43.965068 7940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "62fe781507f347b38cb96307937f891e"
format_stamp: "Formatted at 2024-05-06 15:44:43 on dist-test-slave-v0k1"
server_key: "08ccb1853ef4fde42a749d06f5944a1d"
server_key_iv: "435510bbf03858d9233feaded8b9df9a"
server_key_version: "encryptionkey@0"
I20240506 15:44:43.966240 7940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:44:43.966976 7955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:43.967164 7940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240506 15:44:43.967239 7940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "62fe781507f347b38cb96307937f891e"
format_stamp: "Formatted at 2024-05-06 15:44:43 on dist-test-slave-v0k1"
server_key: "08ccb1853ef4fde42a749d06f5944a1d"
server_key_iv: "435510bbf03858d9233feaded8b9df9a"
server_key_version: "encryptionkey@0"
I20240506 15:44:43.967314 7940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:44.000447 7940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:44.002312 7940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:44.002429 7940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:44.006198 7940 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:42067
I20240506 15:44:44.006237 8007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:42067 every 8 connection(s)
I20240506 15:44:44.006546 7940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:44.007403 8008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:44.009558 8008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e: Bootstrap starting.
I20240506 15:44:44.010077 8008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:44.010289 8008 log.cc:824] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:44.010916 8008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e: No bootstrap required, opened a new log
I20240506 15:44:44.012351 8008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe781507f347b38cb96307937f891e" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42067 } }
I20240506 15:44:44.012437 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 7940
I20240506 15:44:44.012483 8008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:44.012504 8008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fe781507f347b38cb96307937f891e, State: Initialized, Role: FOLLOWER
I20240506 15:44:44.012542 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:44.012653 8008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe781507f347b38cb96307937f891e" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42067 } }
I20240506 15:44:44.012727 8008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:44.012722 18371 external_mini_cluster.cc:1428] Setting key 22e69baf14ded7ce005eb72cdfbe6037
I20240506 15:44:44.012754 8008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:44.012804 8008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:44.013521 8008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe781507f347b38cb96307937f891e" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42067 } }
I20240506 15:44:44.013608 8008 leader_election.cc:302] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62fe781507f347b38cb96307937f891e; no voters:
I20240506 15:44:44.013753 8008 leader_election.cc:288] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:44.013978 8013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:44.014015 8008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:44.014150 8013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [term 1 LEADER]: Becoming Leader. State: Replica: 62fe781507f347b38cb96307937f891e, State: Running, Role: LEADER
I20240506 15:44:44.014290 8013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe781507f347b38cb96307937f891e" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42067 } }
I20240506 15:44:44.014616 8014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62fe781507f347b38cb96307937f891e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe781507f347b38cb96307937f891e" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42067 } } }
I20240506 15:44:44.014708 8014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:44.014878 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fe781507f347b38cb96307937f891e. Latest consensus state: current_term: 1 leader_uuid: "62fe781507f347b38cb96307937f891e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe781507f347b38cb96307937f891e" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 42067 } } }
I20240506 15:44:44.014933 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:44.014955 8022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:44.015417 8022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:44.017160 8022 catalog_manager.cc:1347] Generated new cluster ID: 11469be4b079448aadae9d960f11edb0
I20240506 15:44:44.017215 8022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:44.017654 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:44.013188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55769 (local address 127.17.240.254:42067)
0506 15:44:44.013362 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:44.013375 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:44.013416 (+ 41us) server_negotiation.cc:407] Connection header received
0506 15:44:44.013500 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:44.013504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:44.013541 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:44.013680 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:44.014238 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:44.015623 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:44.016318 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:44.016588 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:44.017002 (+ 414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:44.017008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:44.017008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:44.017206 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:44.017231 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:44.017240 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:44.017319 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:44.017445 (+ 126us) server_negotiation.cc:299] Negotiation successful
0506 15:44:44.017488 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20240506 15:44:44.019944 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:44.034181 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:44.023304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44411 (local address 127.17.240.254:42067)
0506 15:44:44.023353 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:44.023356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:44.031275 (+ 7919us) server_negotiation.cc:407] Connection header received
0506 15:44:44.031367 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:44.031368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:44.031381 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:44.031456 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:44.031807 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:44.032510 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:44.033280 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:44.033446 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:44.033914 (+ 468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:44.033918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:44.033919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:44.033924 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:44.033939 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:44.033943 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:44.033996 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:44.034059 (+ 63us) server_negotiation.cc:299] Negotiation successful
0506 15:44:44.034092 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:44:44.038033 8022 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:44.038741 8022 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:44.046922 8022 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 62fe781507f347b38cb96307937f891e: Generated new TSK 0
I20240506 15:44:44.047171 8022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:44.055168 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 7940
2024-05-06T15:44:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:44 dist-test-slave-v0k1 krb5kdc[8039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:44 dist-test-slave-v0k1 krb5kdc[8039](info): set up 2 sockets
May 06 15:44:44 dist-test-slave-v0k1 krb5kdc[8039](info): commencing operation
krb5kdc: starting...
W20240506 15:44:45.007313 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:45 dist-test-slave-v0k1 krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:45.123528 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:46293
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:46293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:45.206718 8052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:45.206915 8052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:45.206936 8052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:45.208259 8052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:45.208313 8052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:45.208328 8052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:45.208343 8052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:45.208369 8052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:45.209929 8052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:46293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:46293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:45.210131 8052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:45.210316 8052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:45.215101 8060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:45.215569 8052 server_base.cc:1034] running on GCE node
W20240506 15:44:45.216073 8057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:45.216898 8058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:45.217124 8052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:45.217371 8052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:45.218505 8052 hybrid_clock.cc:648] HybridClock initialized: now 1715010285218491 us; error 36 us; skew 500 ppm
May 06 15:44:45 dist-test-slave-v0k1 krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010285, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:45.220928 8052 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:45.221289 8052 webserver.cc:460] Webserver started at http://127.17.240.254:42409/ using document root <none> and password file <none>
I20240506 15:44:45.221443 8052 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:45.221479 8052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:45.221565 8052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:45.222493 8052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "70492f72ab1a417e90434790af694329"
format_stamp: "Formatted at 2024-05-06 15:44:45 on dist-test-slave-v0k1"
server_key: "45fb03be0df4a3655bd1aaad5eb5850b"
server_key_iv: "c56d778018ed91faada0b4a973af7642"
server_key_version: "encryptionkey@0"
I20240506 15:44:45.222751 8052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "70492f72ab1a417e90434790af694329"
format_stamp: "Formatted at 2024-05-06 15:44:45 on dist-test-slave-v0k1"
server_key: "45fb03be0df4a3655bd1aaad5eb5850b"
server_key_iv: "c56d778018ed91faada0b4a973af7642"
server_key_version: "encryptionkey@0"
I20240506 15:44:45.223927 8052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240506 15:44:45.224557 8067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:45.224764 8052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:45.224845 8052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "70492f72ab1a417e90434790af694329"
format_stamp: "Formatted at 2024-05-06 15:44:45 on dist-test-slave-v0k1"
server_key: "45fb03be0df4a3655bd1aaad5eb5850b"
server_key_iv: "c56d778018ed91faada0b4a973af7642"
server_key_version: "encryptionkey@0"
I20240506 15:44:45.224913 8052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:45.228446 8052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:45.230474 8052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:45.230581 8052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:45.234246 8052 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:46293
I20240506 15:44:45.234275 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:46293 every 8 connection(s)
I20240506 15:44:45.234550 8052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:45.235409 8120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:45.237516 8120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329: Bootstrap starting.
I20240506 15:44:45.237674 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 8052
I20240506 15:44:45.237769 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:45.237926 18371 external_mini_cluster.cc:1428] Setting key 6fd1299427de894f71fb8087749faf21
I20240506 15:44:45.238101 8120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:45.238407 8120 log.cc:824] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:45.239171 8120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329: No bootstrap required, opened a new log
I20240506 15:44:45.240494 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:45.238418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53393 (local address 127.17.240.254:46293)
0506 15:44:45.238682 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:45.238693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:45.238709 (+ 16us) server_negotiation.cc:407] Connection header received
0506 15:44:45.239318 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:45.239326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:45.239358 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:45.239517 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:45.240129 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:45.240132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:45.240132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:45.240136 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:45.240159 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:45.240166 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:45.240251 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:45.240286 (+ 35us) server_negotiation.cc:299] Negotiation successful
0506 15:44:45.240318 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":86,"threads_started":1}
I20240506 15:44:45.241190 8120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70492f72ab1a417e90434790af694329" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46293 } }
I20240506 15:44:45.241310 8120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:44:45.241333 8120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70492f72ab1a417e90434790af694329, State: Initialized, Role: FOLLOWER
I20240506 15:44:45.241410 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:45.241439 8120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70492f72ab1a417e90434790af694329" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46293 } }
I20240506 15:44:45.241514 8120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:44:45.241549 8120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:44:45.241590 8120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:45.242318 8120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70492f72ab1a417e90434790af694329" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46293 } }
I20240506 15:44:45.242434 8120 leader_election.cc:302] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70492f72ab1a417e90434790af694329; no voters:
I20240506 15:44:45.242612 8120 leader_election.cc:288] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:44:45.242666 8127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:45.242839 8120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:45.242871 8127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [term 1 LEADER]: Becoming Leader. State: Replica: 70492f72ab1a417e90434790af694329, State: Running, Role: LEADER
I20240506 15:44:45.243148 8127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70492f72ab1a417e90434790af694329" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46293 } }
I20240506 15:44:45.243482 8130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70492f72ab1a417e90434790af694329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70492f72ab1a417e90434790af694329" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46293 } } }
I20240506 15:44:45.243494 8131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70492f72ab1a417e90434790af694329. Latest consensus state: current_term: 1 leader_uuid: "70492f72ab1a417e90434790af694329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70492f72ab1a417e90434790af694329" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 46293 } } }
I20240506 15:44:45.243579 8130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:45.243875 8138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:45.243582 8131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329 [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:45.244376 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:45.243095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57897 (local address 127.17.240.254:46293)
0506 15:44:45.243137 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:45.243138 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:45.243254 (+ 116us) server_negotiation.cc:407] Connection header received
0506 15:44:45.243338 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:45.243340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:45.243346 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:45.243411 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:45.244141 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:45.244142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:45.244143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0506 15:44:45.244147 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:45.244164 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0506 15:44:45.244168 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:45.244217 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:45.244300 (+ 83us) server_negotiation.cc:299] Negotiation successful
0506 15:44:45.244315 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:44:45.244527 8138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:45.246194 8138 catalog_manager.cc:1347] Generated new cluster ID: 2f81cee9cecf465da9dd620a0647edba
I20240506 15:44:45.246241 8138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:45.251837 8138 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:45.252005 8138 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:45.254904 8138 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 70492f72ab1a417e90434790af694329: Generated new TSK 0
I20240506 15:44:45.255110 8138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:44:45.259611 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 8052
2024-05-06T15:44:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1204 ms)
[----------] 18 tests from AuthTokenIssuingTest (22127 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:45 dist-test-slave-v0k1 krb5kdc[8151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:45 dist-test-slave-v0k1 krb5kdc[8151](info): set up 2 sockets
May 06 15:44:45 dist-test-slave-v0k1 krb5kdc[8151](info): commencing operation
krb5kdc: starting...
W20240506 15:44:46.211418 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:46.327574 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:33869
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:33869,127.17.240.253:43129,127.17.240.252:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:46.407238 8164 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 15:44:46.407462 8164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:46.407480 8164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:46.407505 8164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:46.408799 8164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:46.408838 8164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:46.408851 8164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:46.408864 8164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:46.408875 8164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:46.410324 8164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:33869,127.17.240.253:43129,127.17.240.252:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:33869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:46.410490 8164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:46.410637 8164 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:46.415300 8169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:46.415722 8172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:46.416512 8170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:46.417140 8164 server_base.cc:1034] running on GCE node
I20240506 15:44:46.417331 8164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:46.417547 8164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:46.418678 8164 hybrid_clock.cc:648] HybridClock initialized: now 1715010286418664 us; error 32 us; skew 500 ppm
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:46.421208 8164 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:46.421643 8164 webserver.cc:460] Webserver started at http://127.17.240.254:41059/ using document root <none> and password file <none>
I20240506 15:44:46.421818 8164 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:46.421866 8164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:46.421990 8164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:46.423046 8164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "bd1bf284846947f890589ef209270e2f"
format_stamp: "Formatted at 2024-05-06 15:44:46 on dist-test-slave-v0k1"
server_key: "def531a067dfc7f05f9f43566d64d58a"
server_key_iv: "2bab0f19c6c1ac0d430c6c11b140e514"
server_key_version: "encryptionkey@0"
I20240506 15:44:46.423360 8164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "bd1bf284846947f890589ef209270e2f"
format_stamp: "Formatted at 2024-05-06 15:44:46 on dist-test-slave-v0k1"
server_key: "def531a067dfc7f05f9f43566d64d58a"
server_key_iv: "2bab0f19c6c1ac0d430c6c11b140e514"
server_key_version: "encryptionkey@0"
I20240506 15:44:46.424513 8164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:44:46.425166 8179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:46.425330 8164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:46.425407 8164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "bd1bf284846947f890589ef209270e2f"
format_stamp: "Formatted at 2024-05-06 15:44:46 on dist-test-slave-v0k1"
server_key: "def531a067dfc7f05f9f43566d64d58a"
server_key_iv: "2bab0f19c6c1ac0d430c6c11b140e514"
server_key_version: "encryptionkey@0"
I20240506 15:44:46.425479 8164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:46.462093 8164 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:46.464315 8164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:46.464445 8164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:46.468226 8164 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:33869
I20240506 15:44:46.468266 8231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:33869 every 8 connection(s)
I20240506 15:44:46.468591 8164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:46.469434 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:46.470831 8232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:46.472597 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 8164
I20240506 15:44:46.472687 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:46.472877 18371 external_mini_cluster.cc:1428] Setting key f4df1b8a4df5edda75b5697c474effa0
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.240.253@KRBTEST.COM; defaulting to no policy
I20240506 15:44:46.481525 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.471559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:33869 (local address 127.0.0.1:39368)
0506 15:44:46.471704 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.471729 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:46.472191 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:46.472437 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:46.472439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:46.472793 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:46.473062 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.473066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.474371 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.474373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:46.474965 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.474967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.476256 (+ 1289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.476941 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:46.476965 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:46.478345 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:46.480303 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.480309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.480312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.480501 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.480670 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.480671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.480672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.480722 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.481017 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:46.481021 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:46.481081 (+ 60us) client_negotiation.cc:769] Sending connection context
0506 15:44:46.481167 (+ 86us) client_negotiation.cc:240] Negotiation successful
0506 15:44:46.481389 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":83,"thread_start_us":67,"threads_started":1}
I20240506 15:44:46.481526 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.471699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.17.240.254:33869)
0506 15:44:46.471812 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.471822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.471845 (+ 23us) server_negotiation.cc:407] Connection header received
0506 15:44:46.472314 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.472316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.472338 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.472427 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.473162 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.474254 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.476186 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.476443 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.478426 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.478435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.478435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.478441 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.480233 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.480575 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.480577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.480578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.480616 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.480777 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.480777 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.480778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.480959 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.481037 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.481194 (+ 157us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.481319 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":45,"threads_started":1}
I20240506 15:44:46.482199 8232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:46.482539 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.482424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:43129 (local address 127.0.0.1:57480)
0506 15:44:46.482469 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.482514 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:43129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240506 15:44:46.482656 8182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.240.253:43129: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.253@KRBTEST.COM" created.
I20240506 15:44:46.483114 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.482976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:43129 (local address 127.0.0.1:57492)
0506 15:44:46.483044 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.483079 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:43129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20240506 15:44:46.483245 8232 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.253:43129: Network error: Client connection negotiation failed: client connection to 127.17.240.253:43129: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab.
Entry for principal kudu/127.17.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab.
I20240506 15:44:46.498890 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.253:43129
--webserver_interface=127.17.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab
--principal=kudu/127.17.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:33869,127.17.240.253:43129,127.17.240.252:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:44:46.532457 8232 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } attempt: 1
I20240506 15:44:46.532948 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.532816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:43129 (local address 127.0.0.1:57506)
0506 15:44:46.532882 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.532913 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:43129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20240506 15:44:46.533378 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.533295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:43129 (local address 127.0.0.1:57510)
0506 15:44:46.533335 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.533358 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:43129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240506 15:44:46.533473 8232 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.253:43129: Network error: Client connection negotiation failed: client connection to 127.17.240.253:43129: connect: Connection refused (error 111)
W20240506 15:44:46.578140 8239 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 15:44:46.578342 8239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:46.578359 8239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:46.578380 8239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:46.579710 8239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:46.579751 8239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:46.579764 8239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:46.579777 8239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:46.579797 8239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:46.581202 8239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:33869,127.17.240.253:43129,127.17.240.252:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab
--principal=kudu/127.17.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.253:43129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:46.581372 8239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:46.581527 8239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:46.585932 8247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:46.586673 8244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:46.587231 8245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:46.587821 8239 server_base.cc:1034] running on GCE node
I20240506 15:44:46.588006 8239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:46.588207 8239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:46.589332 8239 hybrid_clock.cc:648] HybridClock initialized: now 1715010286589311 us; error 36 us; skew 500 ppm
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:46.591722 8239 init.cc:377] Logged in from keytab as kudu/127.17.240.253@KRBTEST.COM (short username kudu)
I20240506 15:44:46.592118 8239 webserver.cc:460] Webserver started at http://127.17.240.253:41649/ using document root <none> and password file <none>
I20240506 15:44:46.592298 8239 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:46.592341 8239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:46.592455 8239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:46.593423 8239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/data/instance:
uuid: "5417aa18eb9a4057aa30d7934a0ca72e"
format_stamp: "Formatted at 2024-05-06 15:44:46 on dist-test-slave-v0k1"
server_key: "046d772ccabfef1d4e06823bb4e17248"
server_key_iv: "5ff9fe57189ec11dc2f011fcb61fa7cb"
server_key_version: "encryptionkey@0"
I20240506 15:44:46.593732 8239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/wal/instance:
uuid: "5417aa18eb9a4057aa30d7934a0ca72e"
format_stamp: "Formatted at 2024-05-06 15:44:46 on dist-test-slave-v0k1"
server_key: "046d772ccabfef1d4e06823bb4e17248"
server_key_iv: "5ff9fe57189ec11dc2f011fcb61fa7cb"
server_key_version: "encryptionkey@0"
I20240506 15:44:46.594888 8239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240506 15:44:46.595602 8254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:46.595757 8239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:46.595829 8239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/wal
uuid: "5417aa18eb9a4057aa30d7934a0ca72e"
format_stamp: "Formatted at 2024-05-06 15:44:46 on dist-test-slave-v0k1"
server_key: "046d772ccabfef1d4e06823bb4e17248"
server_key_iv: "5ff9fe57189ec11dc2f011fcb61fa7cb"
server_key_version: "encryptionkey@0"
I20240506 15:44:46.595885 8239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:46.601626 8232 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } attempt: 2
I20240506 15:44:46.602052 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.601909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:43129 (local address 127.0.0.1:57516)
0506 15:44:46.601965 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.602006 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:43129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240506 15:44:46.602360 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.602295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:43129 (local address 127.0.0.1:57532)
0506 15:44:46.602324 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.602344 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:43129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20240506 15:44:46.602476 8232 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.253:43129: Network error: Client connection negotiation failed: client connection to 127.17.240.253:43129: connect: Connection refused (error 111)
I20240506 15:44:46.606742 8239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:46.608502 8239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:46.608618 8239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:46.612263 8239 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.253:43129
I20240506 15:44:46.612303 8306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.253:43129 every 8 connection(s)
I20240506 15:44:46.612594 8239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
I20240506 15:44:46.613328 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 8239
I20240506 15:44:46.613479 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-1/wal/instance
I20240506 15:44:46.613554 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:46.613662 18371 external_mini_cluster.cc:1428] Setting key 2e475d06e095c537642ca8119ecb5862
I20240506 15:44:46.615599 8307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.240.252@KRBTEST.COM; defaulting to no policy
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.252@KRBTEST.COM" created.
I20240506 15:44:46.623462 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.616249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.17.240.254:33869)
0506 15:44:46.616309 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.616311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.616477 (+ 166us) server_negotiation.cc:407] Connection header received
0506 15:44:46.617026 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.617028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.617041 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.617105 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.617855 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.618340 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.618862 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.619071 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.620882 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.620887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.620887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.620897 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.622303 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.622665 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.622665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.622666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.622689 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.622862 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.622862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.622863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.622990 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.623080 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.623326 (+ 246us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.623397 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:44:46.623775 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.616196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:33869 (local address 127.0.0.1:39382)
0506 15:44:46.616377 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.616401 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:46.616918 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:46.617167 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:46.617170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:46.617452 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:46.617760 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.617765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.618427 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.618428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:46.618786 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.618787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.618940 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.619518 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:46.619533 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:46.620790 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:46.622383 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.622388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.622390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.622602 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.622746 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.622747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.622748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.622809 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.623070 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:46.623075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:46.623180 (+ 105us) client_negotiation.cc:769] Sending connection context
0506 15:44:46.623296 (+ 116us) client_negotiation.cc:240] Negotiation successful
0506 15:44:46.623448 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20240506 15:44:46.624414 8307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
I20240506 15:44:46.631577 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.624726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:43129 (local address 127.0.0.1:57546)
0506 15:44:46.624780 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.624790 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:46.624835 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:46.625097 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:46.625098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:46.625209 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:46.625344 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.625345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.626586 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.626587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:46.626926 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.626927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.626989 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.627414 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:46.627418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:46.628784 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:46.630827 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.630828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.630828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.630959 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.631139 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.631140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.631140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.631163 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.631339 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:46.631340 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:46.631370 (+ 30us) client_negotiation.cc:769] Sending connection context
0506 15:44:46.631415 (+ 45us) client_negotiation.cc:240] Negotiation successful
0506 15:44:46.631514 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20240506 15:44:46.631691 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.624788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.17.240.253:43129)
0506 15:44:46.624954 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.624964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.624980 (+ 16us) server_negotiation.cc:407] Connection header received
0506 15:44:46.625002 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.625004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.625017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.625092 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.625417 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.626501 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.627030 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.627280 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.628869 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.628878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.628879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.628886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.630750 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.631061 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.631063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.631063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.631089 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.631197 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.631198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.631198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.631283 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.631357 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.631444 (+ 87us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.631579 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20240506 15:44:46.632205 8307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:46.632508 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.632393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:56998)
0506 15:44:46.632434 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.632491 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20240506 15:44:46.632656 8256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
I20240506 15:44:46.633097 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.632975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57012)
0506 15:44:46.633040 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.633070 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20240506 15:44:46.633212 8307 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:37931: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab.
Entry for principal kudu/127.17.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab.
I20240506 15:44:46.639653 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.252:37931
--webserver_interface=127.17.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab
--principal=kudu/127.17.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:33869,127.17.240.253:43129,127.17.240.252:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:44:46.682439 8307 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } attempt: 1
I20240506 15:44:46.682984 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.682839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57018)
0506 15:44:46.682907 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.682941 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20240506 15:44:46.683320 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.683239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57022)
0506 15:44:46.683276 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.683301 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20240506 15:44:46.683401 8307 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:37931: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
I20240506 15:44:46.693622 8232 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } attempt: 3
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
I20240506 15:44:46.700667 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.694004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:43129 (local address 127.0.0.1:57550)
0506 15:44:46.694069 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.694079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:46.694130 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:46.694266 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:46.694266 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:46.694438 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:46.694596 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.694598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.695246 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.695249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:46.695610 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.695611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.695680 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.696260 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:46.696265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:46.698117 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:46.699827 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.699828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.699829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.699985 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.700124 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.700125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.700125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.700151 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.700345 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:46.700347 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:46.700393 (+ 46us) client_negotiation.cc:769] Sending connection context
0506 15:44:46.700451 (+ 58us) client_negotiation.cc:240] Negotiation successful
0506 15:44:46.700562 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20240506 15:44:46.700667 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.694072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.17.240.253:43129)
0506 15:44:46.694134 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.694137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.694168 (+ 31us) server_negotiation.cc:407] Connection header received
0506 15:44:46.694197 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.694198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.694208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.694253 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.694682 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.695169 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.695683 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.695883 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.698205 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.698210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.698210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.698217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.699747 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.700047 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.700048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.700048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.700071 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.700198 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.700199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.700200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.700291 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.700350 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.700485 (+ 135us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.700571 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240506 15:44:46.701054 8232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:46.701445 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.701283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57026)
0506 15:44:46.701351 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.701400 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20240506 15:44:46.701850 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.701754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57040)
0506 15:44:46.701802 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.701827 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20240506 15:44:46.701956 8232 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:37931: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
W20240506 15:44:46.719663 8314 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 15:44:46.719861 8314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:46.719877 8314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:46.719895 8314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:46.721205 8314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:46.721244 8314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:46.721258 8314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:46.721271 8314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:46.721294 8314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:46.722767 8314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:33869,127.17.240.253:43129,127.17.240.252:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab
--principal=kudu/127.17.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.252:37931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:46.722959 8314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:46.723161 8314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:46.727850 8319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:46.728577 8322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:46.729175 8320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:46.729763 8314 server_base.cc:1034] running on GCE node
I20240506 15:44:46.730010 8314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:46.730264 8314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:46.731405 8314 hybrid_clock.cc:648] HybridClock initialized: now 1715010286731388 us; error 35 us; skew 500 ppm
I20240506 15:44:46.733121 8232 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } attempt: 1
I20240506 15:44:46.733479 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.733362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57048)
0506 15:44:46.733419 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.733448 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:46.733803 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.733715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57062)
0506 15:44:46.733757 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.733782 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20240506 15:44:46.733914 8232 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:37931: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
I20240506 15:44:46.733909 8314 init.cc:377] Logged in from keytab as kudu/127.17.240.252@KRBTEST.COM (short username kudu)
I20240506 15:44:46.734331 8314 webserver.cc:460] Webserver started at http://127.17.240.252:33701/ using document root <none> and password file <none>
I20240506 15:44:46.734527 8314 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:46.734572 8314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:46.734678 8314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:46.735703 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/data/instance:
uuid: "e5293a7209d44eca829b4c88c1a52069"
format_stamp: "Formatted at 2024-05-06 15:44:46 on dist-test-slave-v0k1"
server_key: "95f91df7a0ea051236af9d36a52adf35"
server_key_iv: "f27e54deb3eee7fe319781d8657fd7ac"
server_key_version: "encryptionkey@0"
I20240506 15:44:46.736009 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/wal/instance:
uuid: "e5293a7209d44eca829b4c88c1a52069"
format_stamp: "Formatted at 2024-05-06 15:44:46 on dist-test-slave-v0k1"
server_key: "95f91df7a0ea051236af9d36a52adf35"
server_key_iv: "f27e54deb3eee7fe319781d8657fd7ac"
server_key_version: "encryptionkey@0"
I20240506 15:44:46.737208 8314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:44:46.737840 8329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:46.738001 8314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:46.738076 8314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/wal
uuid: "e5293a7209d44eca829b4c88c1a52069"
format_stamp: "Formatted at 2024-05-06 15:44:46 on dist-test-slave-v0k1"
server_key: "95f91df7a0ea051236af9d36a52adf35"
server_key_iv: "f27e54deb3eee7fe319781d8657fd7ac"
server_key_version: "encryptionkey@0"
I20240506 15:44:46.738137 8314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:46.751547 8307 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } attempt: 2
I20240506 15:44:46.751982 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.751863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57072)
0506 15:44:46.751924 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.751952 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20240506 15:44:46.752353 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.752254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57076)
0506 15:44:46.752298 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.752326 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240506 15:44:46.752480 8307 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:37931: Network error: Client connection negotiation failed: client connection to 127.17.240.252:37931: connect: Connection refused (error 111)
I20240506 15:44:46.753501 8314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:46.755245 8314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:46.755342 8314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:46.758885 8314 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.252:37931
I20240506 15:44:46.758944 8381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.252:37931 every 8 connection(s)
I20240506 15:44:46.759245 8314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
I20240506 15:44:46.760107 8382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:46.761510 8382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:46.763943 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 8314
I20240506 15:44:46.764042 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715009978772892-18371-0/minicluster-data/master-2/wal/instance
I20240506 15:44:46.764201 18371 external_mini_cluster.cc:1428] Setting key bfd337dd8ac02f381c85b71c8f00f51f
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
I20240506 15:44:46.770067 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.762177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.17.240.254:33869)
0506 15:44:46.762255 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.762257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.762391 (+ 134us) server_negotiation.cc:407] Connection header received
0506 15:44:46.762894 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.762895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.762907 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.762977 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.763737 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.764431 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.765153 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.765360 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.767339 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.767343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.767344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.767351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.768750 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.769154 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.769155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.769156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.769177 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.769376 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.769377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.769377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.769496 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.769567 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.769857 (+ 290us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.769942 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240506 15:44:46.770174 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.762117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:33869 (local address 127.0.0.1:39390)
0506 15:44:46.762290 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.762315 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:46.762790 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:46.763056 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:46.763059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:46.763355 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:46.763637 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.763641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.764540 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.764541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:46.765072 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.765074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.765172 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.765741 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:46.765756 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:46.767249 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:46.768833 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.768838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.768840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.769079 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.769235 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.769235 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.769236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.769313 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.769556 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:46.769559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:46.769713 (+ 154us) client_negotiation.cc:769] Sending connection context
0506 15:44:46.769847 (+ 134us) client_negotiation.cc:240] Negotiation successful
0506 15:44:46.769977 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20240506 15:44:46.770756 8382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:46.772513 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.764577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.17.240.252:37931)
0506 15:44:46.764720 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.764727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.764774 (+ 47us) server_negotiation.cc:407] Connection header received
0506 15:44:46.764823 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.764825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.764841 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.764928 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.765654 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.766337 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.766906 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.767107 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.769064 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.769072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.769073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.769079 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.771520 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.771815 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.771818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.771818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.771844 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.771989 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.771990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.771990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.772086 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.772217 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.772317 (+ 100us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.772400 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:46.777098 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.771096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.17.240.253:43129)
0506 15:44:46.771151 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.771153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.771160 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:46.771182 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.771183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.771190 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.771235 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.771762 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.772188 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.772714 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.772889 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.774460 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.774463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.774463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.774469 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.776188 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.776526 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.776527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.776527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.776552 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.776707 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.776707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.776707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.776786 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.776831 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.776974 (+ 143us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.777036 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240506 15:44:46.777709 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.771031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:43129 (local address 127.0.0.1:57558)
0506 15:44:46.771065 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.771075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:46.771118 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:46.771239 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:46.771240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:46.771508 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:46.771682 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.771683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.772254 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.772255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:46.772642 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.772644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.772708 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.773129 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:46.773133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:46.774368 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:46.776273 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.776274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.776275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.776448 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.776617 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.776618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.776618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.776651 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.776840 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:46.776841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:46.776888 (+ 47us) client_negotiation.cc:769] Sending connection context
0506 15:44:46.776958 (+ 70us) client_negotiation.cc:240] Negotiation successful
0506 15:44:46.777055 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20240506 15:44:46.778129 8382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:46.780189 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.773305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.17.240.254:33869)
0506 15:44:46.773348 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.773350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.773357 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:46.773415 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.773416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.773424 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.773461 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.774048 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.774481 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.775097 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.775215 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.777647 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.777649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.777650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.777655 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.779322 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.779582 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.779583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.779583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.779604 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.779733 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.779734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.779734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.779815 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.779937 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.780002 (+ 65us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.780077 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
I20240506 15:44:46.784209 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.778382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57092)
0506 15:44:46.778465 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.778474 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:46.778531 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:46.778676 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:46.778677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:46.778823 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:46.779031 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.779033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.779580 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.779581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:46.779878 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.779879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.779941 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.780474 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:46.780478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:46.781644 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:46.783317 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.783318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.783319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.783515 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.783648 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.783648 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.783649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.783679 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.783860 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:46.783861 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:46.783896 (+ 35us) client_negotiation.cc:769] Sending connection context
0506 15:44:46.783956 (+ 60us) client_negotiation.cc:240] Negotiation successful
0506 15:44:46.784063 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20240506 15:44:46.784261 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.778425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.17.240.252:37931)
0506 15:44:46.778458 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.778460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.778506 (+ 46us) server_negotiation.cc:407] Connection header received
0506 15:44:46.778605 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.778606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.778616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.778671 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.779099 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.779498 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.779942 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.780133 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.781722 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.781725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.781725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.781732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.783240 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.783582 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.783583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.783583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.783598 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.783729 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.783730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.783730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.783799 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.783853 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.783980 (+ 127us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.784128 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":17}
I20240506 15:44:46.786386 8382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069: Bootstrap starting.
I20240506 15:44:46.787130 8382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:46.787216 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.780826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.17.240.253:43129)
0506 15:44:46.780871 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.780873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.780883 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:44:46.780903 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.780903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.780913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.780969 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.781477 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.781975 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.782536 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.782714 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.784709 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.784712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.784712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.784719 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.786283 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.786593 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.786594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.786594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.786620 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.786744 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.786745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.786745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.786850 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.786973 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.787033 (+ 60us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.787124 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240506 15:44:46.787434 8382 log.cc:824] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:46.788244 8382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069: No bootstrap required, opened a new log
I20240506 15:44:46.788625 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:46.789880 8382 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } }
I20240506 15:44:46.789989 8382 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5293a7209d44eca829b4c88c1a52069, State: Initialized, Role: FOLLOWER
I20240506 15:44:46.789453 8206 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bd1bf284846947f890589ef209270e2f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400
I20240506 15:44:46.790045 8382 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } }
I20240506 15:44:46.790330 8388 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } } }
I20240506 15:44:46.790366 8382 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:46.790421 8388 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [sys.catalog]: This master's current role is: FOLLOWER
W20240506 15:44:46.792271 8399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:46.792325 8399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:46.792584 8314 master_runner.cc:186] Error getting master registration for 127.17.240.254:33869: OK, instance_id { permanent_uuid: "bd1bf284846947f890589ef209270e2f" instance_seqno: 1715010286467192 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240506 15:44:46.794976 8314 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 15:44:46.809103 8232 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } attempt: 2
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
I20240506 15:44:46.816056 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.809545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.17.240.252:37931)
0506 15:44:46.809600 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.809602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.809612 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:44:46.809638 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.809639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.809649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.809724 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.810106 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.810782 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.811573 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.811758 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.813643 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.813646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.813647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.813653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.814993 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.815356 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.815357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.815357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.815375 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.815523 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.815524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.815525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.815662 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.815733 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.815880 (+ 147us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.815958 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240506 15:44:46.816059 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.809427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57106)
0506 15:44:46.809490 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.809501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:46.809549 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:46.809735 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:46.809735 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:46.809883 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:46.810035 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.810036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.810879 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.810881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:46.811491 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.811494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.811566 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.812145 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:46.812151 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:46.813550 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:46.815104 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.815106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.815106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.815292 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.815411 (+ 119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.815412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.815412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.815455 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.815727 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:46.815728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:46.815777 (+ 49us) client_negotiation.cc:769] Sending connection context
0506 15:44:46.815849 (+ 72us) client_negotiation.cc:240] Negotiation successful
0506 15:44:46.815950 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20240506 15:44:46.817427 8232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f: Bootstrap starting.
I20240506 15:44:46.817998 8232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:46.818250 8232 log.cc:824] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:46.818928 8232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f: No bootstrap required, opened a new log
I20240506 15:44:46.819799 8232 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } }
I20240506 15:44:46.819898 8232 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1bf284846947f890589ef209270e2f, State: Initialized, Role: FOLLOWER
I20240506 15:44:46.819965 8232 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } }
I20240506 15:44:46.820238 8232 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:46.820266 8206 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:44:46.820340 8206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:46.821162 8206 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } }
I20240506 15:44:46.820197 8402 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } } }
I20240506 15:44:46.821871 8402 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [sys.catalog]: This master's current role is: FOLLOWER
W20240506 15:44:46.822181 8413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:46.822269 8413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:46.822649 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd1bf284846947f890589ef209270e2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5293a7209d44eca829b4c88c1a52069"
I20240506 15:44:46.822767 8356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:46.823676 8356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1bf284846947f890589ef209270e2f in term 1.
I20240506 15:44:46.823886 8181 leader_election.cc:302] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd1bf284846947f890589ef209270e2f, e5293a7209d44eca829b4c88c1a52069; no voters:
I20240506 15:44:46.824010 8402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:46.824175 8402 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [term 1 LEADER]: Becoming Leader. State: Replica: bd1bf284846947f890589ef209270e2f, State: Running, Role: LEADER
I20240506 15:44:46.824270 8402 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } }
I20240506 15:44:46.824705 8415 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd1bf284846947f890589ef209270e2f. Latest consensus state: current_term: 1 leader_uuid: "bd1bf284846947f890589ef209270e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } } }
I20240506 15:44:46.824927 8415 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:46.825306 8417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:46.825596 8417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:46.824126 8164 master_runner.cc:186] Error getting master registration for 127.17.240.253:43129: OK, instance_id { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" instance_seqno: 1715010286611194 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240506 15:44:46.826117 8164 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 15:44:46.826171 8206 leader_election.cc:288] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [CANDIDATE]: Term 1 election: Requested vote from peers 5417aa18eb9a4057aa30d7934a0ca72e (127.17.240.253:43129), e5293a7209d44eca829b4c88c1a52069 (127.17.240.252:37931)
I20240506 15:44:46.827639 8356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [term 1 FOLLOWER]: Refusing update from remote peer bd1bf284846947f890589ef209270e2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:46.828056 8415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:46.829252 8388 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd1bf284846947f890589ef209270e2f. Latest consensus state: current_term: 1 leader_uuid: "bd1bf284846947f890589ef209270e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } } }
I20240506 15:44:46.829341 8388 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:46.835738 8415 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd1bf284846947f890589ef209270e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } } }
I20240506 15:44:46.835834 8415 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:46.835922 8418 mvcc.cc:205] Tried to move back new op lower bound from 7024682134843678720 to 7024682134832955392. Current Snapshot: MvccSnapshot[applied={T|T < 7024682134843678720}]
I20240506 15:44:46.836030 8388 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd1bf284846947f890589ef209270e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } } }
I20240506 15:44:46.836118 8388 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:46.840345 8426 mvcc.cc:205] Tried to move back new op lower bound from 7024682134843678720 to 7024682134832955392. Current Snapshot: MvccSnapshot[applied={T|T < 7024682134843678720}]
I20240506 15:44:46.843573 8307 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } attempt: 3
I20240506 15:44:46.845862 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.827951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.17.240.252:37931)
0506 15:44:46.827986 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.827987 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.836373 (+ 8386us) server_negotiation.cc:407] Connection header received
0506 15:44:46.836458 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.836459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.836469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.836495 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.837338 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.837974 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.840409 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.840614 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.842532 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.842549 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.842549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.842556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.844210 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.844800 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.844801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.844802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.844826 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.845149 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.845150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.845151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.845271 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.845357 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.845685 (+ 328us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.845771 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:44:46.846215 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.827311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.17.240.254:33869)
0506 15:44:46.827350 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.827352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.835155 (+ 7803us) server_negotiation.cc:407] Connection header received
0506 15:44:46.835249 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.835250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.835262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.835311 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.836296 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.837003 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.837965 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.838148 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.840404 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.840422 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.840423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.840429 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.842424 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.845197 (+ 2773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.845198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.845199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.845225 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.845382 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.845382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.845383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.845488 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.845573 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.846034 (+ 461us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.846116 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:44:46.847666 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.827632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.17.240.253:43129)
0506 15:44:46.827670 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.827672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.835317 (+ 7645us) server_negotiation.cc:407] Connection header received
0506 15:44:46.835387 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.835388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.835401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.835449 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.836713 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.837394 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.838138 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.838324 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.839829 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.839833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.839834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.839841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.844841 (+ 5000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.846907 (+ 2066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.846909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.846909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.846936 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.847132 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.847133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.847134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.847237 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.847301 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.847497 (+ 196us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.847575 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
May 06 15:44:46 dist-test-slave-v0k1 krb5kdc[8151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010286, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
I20240506 15:44:46.851529 8417 catalog_manager.cc:1347] Generated new cluster ID: 89a5ac6a777d475c8a4ae7977dbb1751
I20240506 15:44:46.851584 8417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:46.852973 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.843828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:37931 (local address 127.0.0.1:57136)
0506 15:44:46.843924 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:46.843934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:46.843969 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:46.846185 (+ 2216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:46.846186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:46.846337 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:46.846517 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.846519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.847664 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.847665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:46.848197 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:46.848198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.848267 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.848727 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:46.848731 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:46.850103 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:46.851975 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.851976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.851976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.852166 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.852375 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:46.852376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:46.852376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:46.852401 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:46.852684 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:46.852685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:46.852738 (+ 53us) client_negotiation.cc:769] Sending connection context
0506 15:44:46.852797 (+ 59us) client_negotiation.cc:240] Negotiation successful
0506 15:44:46.852876 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20240506 15:44:46.853344 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:46.844053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.17.240.252:37931)
0506 15:44:46.845923 (+ 1870us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.845925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.845932 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:46.845949 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.845949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.845958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.846007 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.846876 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.847576 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:46.848617 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:46.848802 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:46.850193 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:46.850196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:46.850196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:46.850202 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:46.851857 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.852262 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.852263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.852263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.852289 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:46.852457 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:46.852458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:46.852458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:46.852606 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:46.852693 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:46.852827 (+ 134us) server_negotiation.cc:299] Negotiation successful
0506 15:44:46.852955 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1858,"thread_start_us":85,"threads_started":1}
I20240506 15:44:46.854741 8307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e: Bootstrap starting.
I20240506 15:44:46.855404 8307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:46.855650 8307 log.cc:824] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:46.857574 8307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e: No bootstrap required, opened a new log
I20240506 15:44:46.859354 8307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } }
I20240506 15:44:46.859470 8307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5417aa18eb9a4057aa30d7934a0ca72e, State: Initialized, Role: FOLLOWER
I20240506 15:44:46.859529 8307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } }
I20240506 15:44:46.859797 8307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:46.859920 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd1bf284846947f890589ef209270e2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5417aa18eb9a4057aa30d7934a0ca72e"
I20240506 15:44:46.860318 8280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:46.861622 8280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [term 1 FOLLOWER]: Refusing update from remote peer bd1bf284846947f890589ef209270e2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:46.861943 8415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 15:44:46.862242 8443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:46.862289 8443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:46.863306 8281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1bf284846947f890589ef209270e2f in term 1.
I20240506 15:44:46.866631 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } } }
I20240506 15:44:46.866801 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:46.866885 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd1bf284846947f890589ef209270e2f. Latest consensus state: current_term: 1 leader_uuid: "bd1bf284846947f890589ef209270e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } } }
I20240506 15:44:46.866981 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:46.867079 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd1bf284846947f890589ef209270e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } } }
I20240506 15:44:46.867172 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:46.875488 8446 mvcc.cc:205] Tried to move back new op lower bound from 7024682134843678720 to 7024682134832955392. Current Snapshot: MvccSnapshot[applied={T|T < 7024682134843678720}]
I20240506 15:44:46.888844 8415 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd1bf284846947f890589ef209270e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1bf284846947f890589ef209270e2f" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 33869 } } peers { permanent_uuid: "5417aa18eb9a4057aa30d7934a0ca72e" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 43129 } } peers { permanent_uuid: "e5293a7209d44eca829b4c88c1a52069" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 37931 } } }
I20240506 15:44:46.888969 8415 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:46.890647 8417 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:46.891559 8417 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:46.900405 8417 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bd1bf284846947f890589ef209270e2f: Generated new TSK 0
I20240506 15:44:46.900635 8417 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240506 15:44:46.925112 8235 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:46.923484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.17.240.254:33869)
0506 15:44:46.923530 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.923532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.924444 (+ 912us) server_negotiation.cc:407] Connection header received
0506 15:44:46.924566 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.924567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.924581 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.924611 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.925045 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39424: BlockingRecv error: recv got EOF from 127.0.0.1:39424 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20240506 15:44:46.925947 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:46.923851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.17.240.253:43129)
0506 15:44:46.923890 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.923891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.925451 (+ 1560us) server_negotiation.cc:407] Connection header received
0506 15:44:46.925559 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.925560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.925570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.925599 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.925888 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57592: BlockingRecv error: recv got EOF from 127.0.0.1:57592 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20240506 15:44:46.926800 8430 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:46.924165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.17.240.252:37931)
0506 15:44:46.924212 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.924214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.926199 (+ 1985us) server_negotiation.cc:407] Connection header received
0506 15:44:46.926361 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.926363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.926375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.926410 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.926744 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57152: BlockingRecv error: recv got EOF from 127.0.0.1:57152 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20240506 15:44:46.928905 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:46.928105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.17.240.253:43129)
0506 15:44:46.928148 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.928150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.928366 (+ 216us) server_negotiation.cc:407] Connection header received
0506 15:44:46.928509 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.928510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.928526 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.928556 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.928841 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57602: BlockingRecv error: recv got EOF from 127.0.0.1:57602 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20240506 15:44:46.931998 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:46.931235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.17.240.253:43129)
0506 15:44:46.931269 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.931271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.931528 (+ 257us) server_negotiation.cc:407] Connection header received
0506 15:44:46.931621 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.931622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.931631 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.931658 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.931952 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57608: BlockingRecv error: recv got EOF from 127.0.0.1:57608 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20240506 15:44:46.936025 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:46.935277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.17.240.253:43129)
0506 15:44:46.935310 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.935311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.935550 (+ 239us) server_negotiation.cc:407] Connection header received
0506 15:44:46.935644 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.935645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.935655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.935683 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.935980 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57622: BlockingRecv error: recv got EOF from 127.0.0.1:57622 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20240506 15:44:46.942065 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:46.941215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.17.240.253:43129)
0506 15:44:46.941251 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.941253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.941569 (+ 316us) server_negotiation.cc:407] Connection header received
0506 15:44:46.941664 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.941665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.941676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.941704 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.942016 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57638: BlockingRecv error: recv got EOF from 127.0.0.1:57638 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20240506 15:44:46.952322 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:46.951431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57640 (local address 127.17.240.253:43129)
0506 15:44:46.951482 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.951485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.951764 (+ 279us) server_negotiation.cc:407] Connection header received
0506 15:44:46.951863 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.951865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.951876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.951905 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.952262 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57640: BlockingRecv error: recv got EOF from 127.0.0.1:57640 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20240506 15:44:46.970784 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:46.969801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.17.240.253:43129)
0506 15:44:46.969845 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:46.969847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:46.970160 (+ 313us) server_negotiation.cc:407] Connection header received
0506 15:44:46.970270 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:46.970271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:46.970283 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:46.970312 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:46.970725 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57656: BlockingRecv error: recv got EOF from 127.0.0.1:57656 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20240506 15:44:47.005324 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:47.004333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.17.240.253:43129)
0506 15:44:47.004379 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:47.004381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:47.004711 (+ 330us) server_negotiation.cc:407] Connection header received
0506 15:44:47.004823 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:47.004824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:47.004835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:47.004864 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:47.005258 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57660: BlockingRecv error: recv got EOF from 127.0.0.1:57660 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20240506 15:44:47.071513 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:47.070843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.17.240.253:43129)
0506 15:44:47.070902 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:47.070904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:47.070984 (+ 80us) server_negotiation.cc:407] Connection header received
0506 15:44:47.071093 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:47.071094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:47.071104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:47.071151 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:47.071460 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57672: BlockingRecv error: recv got EOF from 127.0.0.1:57672 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20240506 15:44:47.201419 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:47.200721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.17.240.253:43129)
0506 15:44:47.200804 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:47.200807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:47.200873 (+ 66us) server_negotiation.cc:407] Connection header received
0506 15:44:47.200956 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:47.200957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:47.200968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:47.201028 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:47.201351 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57674: BlockingRecv error: recv got EOF from 127.0.0.1:57674 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20240506 15:44:47.459347 8312 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:47.458717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.17.240.253:43129)
0506 15:44:47.458784 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:47.458786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:47.458796 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:44:47.458868 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:47.458868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:47.458878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:47.458918 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:47.459272 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57682: BlockingRecv error: recv got EOF from 127.0.0.1:57682 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20240506 15:44:47.792747 8399 catalog_manager.cc:1259] Loaded cluster ID: 89a5ac6a777d475c8a4ae7977dbb1751
I20240506 15:44:47.792829 8399 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069: loading cluster ID for follower catalog manager: success
I20240506 15:44:47.793692 8399 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069: acquiring CA information for follower catalog manager: success
I20240506 15:44:47.793958 8399 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5293a7209d44eca829b4c88c1a52069: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 15:44:47.862684 8443 catalog_manager.cc:1259] Loaded cluster ID: 89a5ac6a777d475c8a4ae7977dbb1751
I20240506 15:44:47.862766 8443 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e: loading cluster ID for follower catalog manager: success
I20240506 15:44:47.863618 8443 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e: acquiring CA information for follower catalog manager: success
I20240506 15:44:47.863871 8443 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5417aa18eb9a4057aa30d7934a0ca72e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 15:44:47.974794 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:47.972603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.17.240.253:43129)
0506 15:44:47.972741 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:47.972744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:47.972755 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:44:47.972806 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:47.972807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:47.972853 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:47.972927 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:47.973210 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:47.973706 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:47.974276 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:47.974414 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:47.974454 (+ 40us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:47.974544 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:47.974618 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:47.974676 (+ 58us) server_negotiation.cc:299] Negotiation successful
0506 15:44:47.974707 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20240506 15:44:47.978412 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:47.976529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.17.240.252:37931)
0506 15:44:47.976658 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:47.976660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:47.976668 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:44:47.976707 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:47.976707 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:47.976730 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:47.976803 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:47.976995 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:47.977503 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:47.977989 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:47.978100 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:47.978133 (+ 33us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:47.978214 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:47.978273 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:47.978291 (+ 18us) server_negotiation.cc:299] Negotiation successful
0506 15:44:47.978317 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20240506 15:44:47.982523 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:47.980263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.17.240.253:43129)
0506 15:44:47.980325 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:47.980327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:47.980452 (+ 125us) server_negotiation.cc:407] Connection header received
0506 15:44:47.980536 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:47.980537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:47.980549 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:47.980593 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:47.980898 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:47.981352 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:47.982035 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:47.982201 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:47.982221 (+ 20us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:47.982249 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:47.982296 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:47.982403 (+ 107us) server_negotiation.cc:299] Negotiation successful
0506 15:44:47.982431 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240506 15:44:47.982956 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:47.980129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.17.240.252:37931)
0506 15:44:47.980178 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:47.980180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:47.980435 (+ 255us) server_negotiation.cc:407] Connection header received
0506 15:44:47.980489 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:47.980490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:47.980502 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:47.980547 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:47.980865 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:47.981602 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:47.982279 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:47.982432 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:47.982453 (+ 21us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:47.982478 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:47.982535 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:47.982862 (+ 327us) server_negotiation.cc:299] Negotiation successful
0506 15:44:47.982888 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240506 15:44:47.983142 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:47.980085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.17.240.254:33869)
0506 15:44:47.980184 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:47.980186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:47.980195 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:44:47.980334 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:47.980336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:47.980371 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:47.980464 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:47.980825 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:47.981616 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:47.982568 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:47.982743 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:47.982797 (+ 54us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:47.982908 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:47.982967 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:47.983029 (+ 62us) server_negotiation.cc:299] Negotiation successful
0506 15:44:47.983063 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
I20240506 15:44:47.984779 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 8164
I20240506 15:44:47.990154 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 8239
I20240506 15:44:47.995361 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 8314
2024-05-06T15:44:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2737 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 06 15:44:48 dist-test-slave-v0k1 krb5kdc[8534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 06 15:44:48 dist-test-slave-v0k1 krb5kdc[8534](info): set up 2 sockets
May 06 15:44:48 dist-test-slave-v0k1 krb5kdc[8534](info): commencing operation
krb5kdc: starting...
W20240506 15:44:48.959506 18371 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/test-user.keytab.
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-06T15:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T15:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
Entry for principal kudu/127.17.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab.
I20240506 15:44:49.073454 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:37467
--webserver_interface=127.17.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:37467,127.17.240.253:32831,127.17.240.252:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240506 15:44:49.152037 8547 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 15:44:49.152248 8547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:49.152266 8547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:49.152285 8547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:49.153589 8547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:49.153632 8547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:49.153646 8547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:49.153658 8547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:49.153669 8547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:49.155176 8547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:37467,127.17.240.253:32831,127.17.240.252:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.254.keytab
--principal=kudu/127.17.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.254:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:49.155364 8547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:49.155525 8547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:49.160141 8553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:49.160624 8552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:49.161131 8555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:49.161948 8547 server_base.cc:1034] running on GCE node
I20240506 15:44:49.162185 8547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:49.162421 8547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:49.163566 8547 hybrid_clock.cc:648] HybridClock initialized: now 1715010289163554 us; error 30 us; skew 500 ppm
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:49.177101 8547 init.cc:377] Logged in from keytab as kudu/127.17.240.254@KRBTEST.COM (short username kudu)
I20240506 15:44:49.177613 8547 webserver.cc:460] Webserver started at http://127.17.240.254:40359/ using document root <none> and password file <none>
I20240506 15:44:49.177831 8547 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:49.177891 8547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:49.178023 8547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:49.179370 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/data/instance:
uuid: "b18d6697d12e4a0c80d25febc47c0c2a"
format_stamp: "Formatted at 2024-05-06 15:44:49 on dist-test-slave-v0k1"
server_key: "5a771c916ce855bdea7910d771789798"
server_key_iv: "63abb1f6f7ef169b27f285cba2a80d83"
server_key_version: "encryptionkey@0"
I20240506 15:44:49.179709 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/wal/instance:
uuid: "b18d6697d12e4a0c80d25febc47c0c2a"
format_stamp: "Formatted at 2024-05-06 15:44:49 on dist-test-slave-v0k1"
server_key: "5a771c916ce855bdea7910d771789798"
server_key_iv: "63abb1f6f7ef169b27f285cba2a80d83"
server_key_version: "encryptionkey@0"
I20240506 15:44:49.181034 8547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240506 15:44:49.181717 8562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:49.181879 8547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240506 15:44:49.181951 8547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/wal
uuid: "b18d6697d12e4a0c80d25febc47c0c2a"
format_stamp: "Formatted at 2024-05-06 15:44:49 on dist-test-slave-v0k1"
server_key: "5a771c916ce855bdea7910d771789798"
server_key_iv: "63abb1f6f7ef169b27f285cba2a80d83"
server_key_version: "encryptionkey@0"
I20240506 15:44:49.182029 8547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:49.191543 8547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:49.193629 8547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:49.193750 8547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:49.197505 8547 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.254:37467
I20240506 15:44:49.197551 8614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.254:37467 every 8 connection(s)
I20240506 15:44:49.197853 8547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/data/info.pb
I20240506 15:44:49.198657 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:49.200098 8615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:49.207739 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 8547
I20240506 15:44:49.207824 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-0/wal/instance
I20240506 15:44:49.208009 18371 external_mini_cluster.cc:1428] Setting key 705d36bb46c27f97c0533afd5b52bdb2
I20240506 15:44:49.208182 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.200681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:37467 (local address 127.0.0.1:42460)
0506 15:44:49.200818 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.200842 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:49.201332 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:49.201581 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:49.201583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:49.201905 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:49.202140 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.202143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.202953 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.202953 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:49.203387 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.203388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.203522 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.204056 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:49.204073 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:49.205466 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:49.207041 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.207046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.207049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.207220 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.207387 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.207388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.207388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.207432 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.207715 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:49.207718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:49.207777 (+ 59us) client_negotiation.cc:769] Sending connection context
0506 15:44:49.207872 (+ 95us) client_negotiation.cc:240] Negotiation successful
0506 15:44:49.207986 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20240506 15:44:49.208350 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.200818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.17.240.254:37467)
0506 15:44:49.200917 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.200928 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.200950 (+ 22us) server_negotiation.cc:407] Connection header received
0506 15:44:49.201471 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.201472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.201487 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.201568 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.202231 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.202867 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.203467 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.203662 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.205558 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.205565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.205565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.205570 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.206929 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.207293 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.207296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.207296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.207329 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.207485 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.207486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.207486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.207656 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.207733 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.208017 (+ 284us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.208149 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":39,"threads_started":1}
I20240506 15:44:49.209061 8615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:49.209416 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.209277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:32831 (local address 127.0.0.1:40132)
0506 15:44:49.209328 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.209380 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240506 15:44:49.209518 8564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.240.253:32831: connect: Connection refused (error 111)
I20240506 15:44:49.209929 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.209840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:32831 (local address 127.0.0.1:40148)
0506 15:44:49.209882 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.209907 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240506 15:44:49.210016 8615 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.253:32831: Network error: Client connection negotiation failed: client connection to 127.17.240.253:32831: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.17.240.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab.
Entry for principal kudu/127.17.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab.
I20240506 15:44:49.233973 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.253:32831
--webserver_interface=127.17.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab
--principal=kudu/127.17.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:37467,127.17.240.253:32831,127.17.240.252:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:44:49.259249 8615 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } attempt: 1
I20240506 15:44:49.259802 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.259647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:32831 (local address 127.0.0.1:40150)
0506 15:44:49.259717 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.259767 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20240506 15:44:49.260226 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.260130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:32831 (local address 127.0.0.1:40166)
0506 15:44:49.260174 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.260205 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240506 15:44:49.260351 8615 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.253:32831: Network error: Client connection negotiation failed: client connection to 127.17.240.253:32831: connect: Connection refused (error 111)
W20240506 15:44:49.314265 8622 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 15:44:49.314489 8622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:49.314507 8622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:49.314527 8622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:49.315892 8622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:49.315939 8622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:49.315953 8622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:49.315965 8622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:49.315976 8622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:49.317498 8622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:37467,127.17.240.253:32831,127.17.240.252:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.253.keytab
--principal=kudu/127.17.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.253:32831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:49.317713 8622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:49.317919 8622 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:49.322746 8628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:49.323382 8630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:49.323933 8627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:49.324627 8622 server_base.cc:1034] running on GCE node
I20240506 15:44:49.324834 8622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:49.325066 8622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:49.326195 8622 hybrid_clock.cc:648] HybridClock initialized: now 1715010289326179 us; error 38 us; skew 500 ppm
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:49.328531 8615 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } attempt: 2
I20240506 15:44:49.328902 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.328781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:32831 (local address 127.0.0.1:40168)
0506 15:44:49.328844 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.328875 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20240506 15:44:49.328914 8622 init.cc:377] Logged in from keytab as kudu/127.17.240.253@KRBTEST.COM (short username kudu)
I20240506 15:44:49.329221 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.329143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:32831 (local address 127.0.0.1:40174)
0506 15:44:49.329172 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.329198 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.253:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20240506 15:44:49.329313 8615 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.253:32831: Network error: Client connection negotiation failed: client connection to 127.17.240.253:32831: connect: Connection refused (error 111)
I20240506 15:44:49.329367 8622 webserver.cc:460] Webserver started at http://127.17.240.253:42171/ using document root <none> and password file <none>
I20240506 15:44:49.329562 8622 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:49.329602 8622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:49.329691 8622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:49.330641 8622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/data/instance:
uuid: "2648e93499a84392967f35adfc2478c2"
format_stamp: "Formatted at 2024-05-06 15:44:49 on dist-test-slave-v0k1"
server_key: "567bda6ba8c907216d06a8736bb66510"
server_key_iv: "f7f3200370d9a3dfd640ecf7689e5ccf"
server_key_version: "encryptionkey@0"
I20240506 15:44:49.330907 8622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/wal/instance:
uuid: "2648e93499a84392967f35adfc2478c2"
format_stamp: "Formatted at 2024-05-06 15:44:49 on dist-test-slave-v0k1"
server_key: "567bda6ba8c907216d06a8736bb66510"
server_key_iv: "f7f3200370d9a3dfd640ecf7689e5ccf"
server_key_version: "encryptionkey@0"
I20240506 15:44:49.332125 8622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:44:49.332779 8637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:49.332971 8622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:49.333053 8622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/wal
uuid: "2648e93499a84392967f35adfc2478c2"
format_stamp: "Formatted at 2024-05-06 15:44:49 on dist-test-slave-v0k1"
server_key: "567bda6ba8c907216d06a8736bb66510"
server_key_iv: "f7f3200370d9a3dfd640ecf7689e5ccf"
server_key_version: "encryptionkey@0"
I20240506 15:44:49.333104 8622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:49.358701 8622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:49.360508 8622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:49.360621 8622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:49.364440 8622 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.253:32831
I20240506 15:44:49.364495 8689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.253:32831 every 8 connection(s)
I20240506 15:44:49.364795 8622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/data/info.pb
I20240506 15:44:49.365669 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:49.367090 8690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:49.368572 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 8622
I20240506 15:44:49.368691 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-1/wal/instance
I20240506 15:44:49.368862 18371 external_mini_cluster.cc:1428] Setting key 7c51f04182e32d0b472c8259419c4f3a
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:49.375888 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.367820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.17.240.254:37467)
0506 15:44:49.367890 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.367893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.368022 (+ 129us) server_negotiation.cc:407] Connection header received
0506 15:44:49.368532 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.368534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.368544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.368604 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.369630 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.370091 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.370648 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.370836 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.373197 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.373201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.373201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.373208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.374623 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.375097 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.375098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.375098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.375116 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.375328 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.375329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.375329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.375410 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.375453 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.375761 (+ 308us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.375824 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240506 15:44:49.376082 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.367720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:37467 (local address 127.0.0.1:42464)
0506 15:44:49.367905 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.367936 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:49.368410 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:49.368684 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:49.368690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:49.369141 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:49.369527 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.369532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.370193 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.370194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:49.370570 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.370572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.370710 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.371326 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:49.371342 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:49.373064 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:49.374706 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.374712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.374716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.374995 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.375170 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.375171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.375171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.375257 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.375463 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:49.375467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:49.375620 (+ 153us) client_negotiation.cc:769] Sending connection context
0506 15:44:49.375759 (+ 139us) client_negotiation.cc:240] Negotiation successful
0506 15:44:49.375897 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.17.240.252@KRBTEST.COM; defaulting to no policy
I20240506 15:44:49.376622 8690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.240.252@KRBTEST.COM" created.
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
I20240506 15:44:49.383426 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.376937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.17.240.253:32831)
0506 15:44:49.377094 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.377101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.377112 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:44:49.377129 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.377130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.377139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.377240 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.377750 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.378463 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.379036 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.379234 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.380706 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.380715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.380716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.380723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.382381 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.382736 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.382738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.382738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.382766 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.382938 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.382938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.382938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.383062 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.383128 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.383280 (+ 152us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.383351 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20240506 15:44:49.383575 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.376893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:32831 (local address 127.0.0.1:40180)
0506 15:44:49.376948 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.376959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:49.377002 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:49.377324 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:49.377325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:49.377453 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:49.377593 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.377594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.378541 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.378542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:49.378932 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.378934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.379047 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.379547 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:49.379551 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:49.380623 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:49.382468 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.382469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.382470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.382654 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.382826 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.382827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.382827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.382864 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.383119 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:49.383120 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:49.383173 (+ 53us) client_negotiation.cc:769] Sending connection context
0506 15:44:49.383260 (+ 87us) client_negotiation.cc:240] Negotiation successful
0506 15:44:49.383419 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":47}
I20240506 15:44:49.384173 8690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:49.384543 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.384410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45438)
0506 15:44:49.384463 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.384518 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240506 15:44:49.384670 8639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
I20240506 15:44:49.385090 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.384987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45446)
0506 15:44:49.385037 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.385065 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20240506 15:44:49.385207 8690 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:45075: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab.
Entry for principal kudu/127.17.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab.
I20240506 15:44:49.394623 18371 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb5AjvH/build/release/bin/kudu
/tmp/dist-test-taskb5AjvH/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.252:45075
--webserver_interface=127.17.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab
--principal=kudu/127.17.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.240.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.240.254:37467,127.17.240.253:32831,127.17.240.252:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240506 15:44:49.420460 8615 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } attempt: 3
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
I20240506 15:44:49.427634 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.420882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:32831 (local address 127.0.0.1:40194)
0506 15:44:49.420943 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.420957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:49.421021 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:49.421152 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:49.421152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:49.421340 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:49.421573 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.421576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.422150 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.422150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:49.422710 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.422712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.422782 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.423320 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:49.423325 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:49.425185 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:49.426732 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.426733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.426733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.426910 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.427074 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.427075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.427076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.427106 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.427298 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:49.427299 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:49.427346 (+ 47us) client_negotiation.cc:769] Sending connection context
0506 15:44:49.427434 (+ 88us) client_negotiation.cc:240] Negotiation successful
0506 15:44:49.427534 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20240506 15:44:49.427647 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.420972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.17.240.253:32831)
0506 15:44:49.421031 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.421033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.421045 (+ 12us) server_negotiation.cc:407] Connection header received
0506 15:44:49.421075 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.421076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.421087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.421137 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.421648 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.422088 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.422785 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.422898 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.425277 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.425282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.425282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.425288 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.426655 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.426977 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.426978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.426978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.426995 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.427157 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.427157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.427158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.427247 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.427296 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.427451 (+ 155us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.427528 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240506 15:44:49.428169 8615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:49.428509 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.428403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45456)
0506 15:44:49.428447 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.428481 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20240506 15:44:49.428807 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.428733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45472)
0506 15:44:49.428767 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.428790 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240506 15:44:49.428897 8615 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:45075: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
I20240506 15:44:49.434419 8690 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } attempt: 1
I20240506 15:44:49.434810 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.434693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45480)
0506 15:44:49.434747 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.434782 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20240506 15:44:49.435165 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.435083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45486)
0506 15:44:49.435120 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.435145 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20240506 15:44:49.435295 8690 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:45075: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
I20240506 15:44:49.460076 8615 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } attempt: 1
I20240506 15:44:49.460709 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.460545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45502)
0506 15:44:49.460620 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.460666 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20240506 15:44:49.461128 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.461027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45508)
0506 15:44:49.461072 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.461100 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240506 15:44:49.461236 8615 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:45075: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
W20240506 15:44:49.475482 8697 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 15:44:49.475674 8697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 15:44:49.475689 8697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 15:44:49.475709 8697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 15:44:49.477007 8697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 15:44:49.477051 8697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240506 15:44:49.477064 8697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 15:44:49.477075 8697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 15:44:49.477087 8697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 15:44:49.478538 8697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.240.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.240.254:37467,127.17.240.253:32831,127.17.240.252:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/krb5kdc/kudu_127.17.240.252.keytab
--principal=kudu/127.17.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.240.252:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 06 May 2024 15:29:15 UTC on 1a15035a6830
build id 157
I20240506 15:44:49.478734 8697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:44:49.478914 8697 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:44:49.483525 8703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:49.483956 8705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:44:49.484727 8702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:44:49.485433 8697 server_base.cc:1034] running on GCE node
I20240506 15:44:49.485641 8697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 15:44:49.485857 8697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 15:44:49.486992 8697 hybrid_clock.cc:648] HybridClock initialized: now 1715010289486975 us; error 44 us; skew 500 ppm
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240506 15:44:49.489392 8697 init.cc:377] Logged in from keytab as kudu/127.17.240.252@KRBTEST.COM (short username kudu)
I20240506 15:44:49.489763 8697 webserver.cc:460] Webserver started at http://127.17.240.252:34025/ using document root <none> and password file <none>
I20240506 15:44:49.489921 8697 fs_manager.cc:359] Metadata directory not provided
I20240506 15:44:49.489956 8697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:44:49.490044 8697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:44:49.491046 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/data/instance:
uuid: "032bb8b800c641ecb2d62ae84de1ffea"
format_stamp: "Formatted at 2024-05-06 15:44:49 on dist-test-slave-v0k1"
server_key: "e3bcf0a14ab27fc91b2f0d2b96e7136d"
server_key_iv: "4e0f70f8ba6ed1ae0c8fa1b86886d4e9"
server_key_version: "encryptionkey@0"
I20240506 15:44:49.491351 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/wal/instance:
uuid: "032bb8b800c641ecb2d62ae84de1ffea"
format_stamp: "Formatted at 2024-05-06 15:44:49 on dist-test-slave-v0k1"
server_key: "e3bcf0a14ab27fc91b2f0d2b96e7136d"
server_key_iv: "4e0f70f8ba6ed1ae0c8fa1b86886d4e9"
server_key_version: "encryptionkey@0"
I20240506 15:44:49.492547 8697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240506 15:44:49.493211 8712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:44:49.493378 8697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240506 15:44:49.493449 8697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/data,/tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/wal
uuid: "032bb8b800c641ecb2d62ae84de1ffea"
format_stamp: "Formatted at 2024-05-06 15:44:49 on dist-test-slave-v0k1"
server_key: "e3bcf0a14ab27fc91b2f0d2b96e7136d"
server_key_iv: "4e0f70f8ba6ed1ae0c8fa1b86886d4e9"
server_key_version: "encryptionkey@0"
I20240506 15:44:49.493499 8697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:44:49.501945 8697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:44:49.503453 8690 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } attempt: 2
I20240506 15:44:49.503801 8697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:44:49.503845 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.503703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45516)
0506 15:44:49.503767 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.503805 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20240506 15:44:49.503923 8697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:44:49.504235 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.504100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45526)
0506 15:44:49.504162 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.504197 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20240506 15:44:49.504390 8690 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.240.252:45075: Network error: Client connection negotiation failed: client connection to 127.17.240.252:45075: connect: Connection refused (error 111)
I20240506 15:44:49.507606 8697 rpc_server.cc:307] RPC server started. Bound to: 127.17.240.252:45075
I20240506 15:44:49.507647 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.240.252:45075 every 8 connection(s)
I20240506 15:44:49.507926 8697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/data/info.pb
I20240506 15:44:49.508764 8765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:44:49.508816 18371 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb5AjvH/build/release/bin/kudu as pid 8697
I20240506 15:44:49.508926 18371 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb5AjvH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715009978772892-18371-0/minicluster-data/master-2/wal/instance
I20240506 15:44:49.509088 18371 external_mini_cluster.cc:1428] Setting key c996da8b609855e331052701bccd3947
I20240506 15:44:49.510751 8765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:49.518193 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.511262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.254:37467 (local address 127.0.0.1:42476)
0506 15:44:49.511434 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.511460 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:49.511520 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:49.511974 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:49.511976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:49.512256 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:49.512432 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.512434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.513193 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.513194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:49.513524 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.513526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.513595 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.514006 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:49.514019 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:49.515439 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:49.517253 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.517259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.517262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.517469 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.517618 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.517618 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.517618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.517640 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.517852 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:49.517857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:49.517912 (+ 55us) client_negotiation.cc:769] Sending connection context
0506 15:44:49.518016 (+ 104us) client_negotiation.cc:240] Negotiation successful
0506 15:44:49.518114 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20240506 15:44:49.518218 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.511792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.17.240.254:37467)
0506 15:44:49.511854 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.511856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.511864 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:44:49.511885 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.511886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.511894 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.511944 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.512518 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.513105 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.513592 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.513776 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.515540 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.515545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.515546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.515554 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.517183 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.517536 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.517538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.517538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.517561 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.517689 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.517690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.517690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.517792 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.517856 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.518054 (+ 198us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.518127 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240506 15:44:49.518194 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.509532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.17.240.252:45075)
0506 15:44:49.509695 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.509707 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.509728 (+ 21us) server_negotiation.cc:407] Connection header received
0506 15:44:49.510147 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.510152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.510192 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.510310 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.511074 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.511946 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.512528 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.512681 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.514844 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.514851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.514852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.514857 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.516965 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.517266 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.517268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.517268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.517313 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.517466 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.517466 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.517467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.517685 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.517808 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.517913 (+ 105us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.518004 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20240506 15:44:49.518698 8765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } has no permanent_uuid. Determining permanent_uuid...
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.254@KRBTEST.COM
I20240506 15:44:49.524230 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.519029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.17.240.253:32831)
0506 15:44:49.519070 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.519072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.519092 (+ 20us) server_negotiation.cc:407] Connection header received
0506 15:44:49.519114 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.519115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.519123 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.519174 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.519512 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.519927 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.520354 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.520522 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.521935 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.521939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.521939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.521946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.523379 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.523650 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.523652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.523652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.523674 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.523802 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.523803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.523803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.523884 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.523936 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.524065 (+ 129us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.524136 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:44:49.524236 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.518932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.253:32831 (local address 127.0.0.1:40208)
0506 15:44:49.518982 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.518992 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:49.519048 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:49.519178 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:49.519179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:49.519309 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:49.519443 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.519444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.519992 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.519993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:49.520293 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.520294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.520351 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.520809 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:49.520813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:49.521858 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:49.523454 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.523455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.523456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.523590 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.523724 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.523724 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.523725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.523751 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.523931 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:49.523932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:49.523980 (+ 48us) client_negotiation.cc:769] Sending connection context
0506 15:44:49.524040 (+ 60us) client_negotiation.cc:240] Negotiation successful
0506 15:44:49.524128 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20240506 15:44:49.524680 8765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } has no permanent_uuid. Determining permanent_uuid...
I20240506 15:44:49.525782 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.519111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.17.240.254:37467)
0506 15:44:49.519146 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.519147 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.519154 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:49.519171 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.519172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.519179 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.519232 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.519926 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.520324 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.520971 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.521138 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.523202 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.523207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.523207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.523213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.524731 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.525161 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.525163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.525163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.525186 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.525314 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.525315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.525316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.525402 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.525522 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.525607 (+ 85us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.525684 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.252@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.240.253@KRBTEST.COM
I20240506 15:44:49.530202 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.524897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45550)
0506 15:44:49.524949 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.524959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:49.525000 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:49.525119 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:49.525119 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:49.525244 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:49.525376 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.525377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.526109 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.526110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:49.526516 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.526518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.526579 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.527056 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:49.527059 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:49.527991 (+ 932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:49.529412 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.529413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.529413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.529586 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.529757 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.529758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.529758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.529786 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.529969 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:49.529970 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:49.530012 (+ 42us) client_negotiation.cc:769] Sending connection context
0506 15:44:49.530078 (+ 66us) client_negotiation.cc:240] Negotiation successful
0506 15:44:49.530146 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20240506 15:44:49.530424 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.524927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.17.240.252:45075)
0506 15:44:49.524976 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.524978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.524987 (+ 9us) server_negotiation.cc:407] Connection header received
0506 15:44:49.525055 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.525056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.525064 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.525116 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.525447 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.525973 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.526579 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.526751 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.528063 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.528066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.528066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.528071 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.529338 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.529672 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.529673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.529673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.529694 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.529842 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.529843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.529843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.529921 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.529967 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.530295 (+ 328us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.530371 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240506 15:44:49.531958 8765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea: Bootstrap starting.
I20240506 15:44:49.532482 8765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:49.532738 8765 log.cc:824] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:49.532944 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.526425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.17.240.253:32831)
0506 15:44:49.526478 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.526479 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.526486 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:49.526504 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.526505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.526512 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.526565 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.527247 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.527644 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.528171 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.528333 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.530473 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.530476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.530477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.530483 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.532001 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.532285 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.532286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.532286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.532308 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.532444 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.532444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.532445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.532533 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.532663 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.532765 (+ 102us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.532844 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240506 15:44:49.533553 8765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea: No bootstrap required, opened a new log
I20240506 15:44:49.535178 18371 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240506 15:44:49.535482 8765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } }
I20240506 15:44:49.535605 8765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032bb8b800c641ecb2d62ae84de1ffea, State: Initialized, Role: FOLLOWER
I20240506 15:44:49.535682 8765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } }
I20240506 15:44:49.535976 8765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:49.535935 8771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } } }
I20240506 15:44:49.536036 8771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:49.536391 8615 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } attempt: 2
I20240506 15:44:49.536118 8589 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b18d6697d12e4a0c80d25febc47c0c2a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42492
W20240506 15:44:49.537827 8782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:49.537878 8782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:49.537986 8697 master_runner.cc:186] Error getting master registration for 127.17.240.254:37467: OK, instance_id { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" instance_seqno: 1715010289196449 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240506 15:44:49.540339 8697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.254@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
I20240506 15:44:49.543483 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.536668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:45564)
0506 15:44:49.536728 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.536739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:49.536787 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:49.536929 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:49.536930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:49.537140 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:49.537335 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.537338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.538199 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.538200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:49.538711 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.538713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.538783 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.539291 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:49.539297 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:49.540748 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:49.542580 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.542581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.542581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.542723 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.542859 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.542860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.542861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.542886 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.543152 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:49.543153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:49.543212 (+ 59us) client_negotiation.cc:769] Sending connection context
0506 15:44:49.543286 (+ 74us) client_negotiation.cc:240] Negotiation successful
0506 15:44:49.543378 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20240506 15:44:49.543815 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.536770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.17.240.252:45075)
0506 15:44:49.536805 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.536808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.536831 (+ 23us) server_negotiation.cc:407] Connection header received
0506 15:44:49.536850 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.536851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.536860 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.536907 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.537487 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.538080 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.538952 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.539196 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.540854 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.540857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.540857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.540863 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.542510 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.542779 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.542780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.542781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.542803 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.542937 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.542938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.542939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.543095 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.543157 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.543666 (+ 509us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.543743 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:44:49.545266 8615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a: Bootstrap starting.
I20240506 15:44:49.545923 8615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:49.546219 8615 log.cc:824] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:49.547049 8615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a: No bootstrap required, opened a new log
I20240506 15:44:49.548238 8615 raft_consensus.cc:348] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } }
I20240506 15:44:49.548365 8615 raft_consensus.cc:729] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18d6697d12e4a0c80d25febc47c0c2a, State: Initialized, Role: FOLLOWER
I20240506 15:44:49.548434 8615 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } }
I20240506 15:44:49.548718 8785 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } } }
I20240506 15:44:49.548732 8615 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:49.548795 8589 raft_consensus.cc:482] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:44:49.548895 8785 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:49.550360 8589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:49.551570 8589 raft_consensus.cc:504] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } }
I20240506 15:44:49.551832 8589 leader_election.cc:288] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [CANDIDATE]: Term 1 election: Requested vote from peers 2648e93499a84392967f35adfc2478c2 (127.17.240.253:32831), 032bb8b800c641ecb2d62ae84de1ffea (127.17.240.252:45075)
I20240506 15:44:49.552208 8739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "032bb8b800c641ecb2d62ae84de1ffea"
I20240506 15:44:49.552331 8739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:49.553304 8739 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b18d6697d12e4a0c80d25febc47c0c2a in term 1.
I20240506 15:44:49.553532 8564 leader_election.cc:302] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032bb8b800c641ecb2d62ae84de1ffea, b18d6697d12e4a0c80d25febc47c0c2a; no voters:
I20240506 15:44:49.553680 8785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:44:49.553809 8785 raft_consensus.cc:686] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [term 1 LEADER]: Becoming Leader. State: Replica: b18d6697d12e4a0c80d25febc47c0c2a, State: Running, Role: LEADER
I20240506 15:44:49.553890 8785 consensus_queue.cc:237] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } }
I20240506 15:44:49.554272 8785 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18d6697d12e4a0c80d25febc47c0c2a. Latest consensus state: current_term: 1 leader_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } } }
I20240506 15:44:49.554339 8785 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [sys.catalog]: This master's current role is: LEADER
W20240506 15:44:49.555064 8800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 15:44:49.555157 8800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 15:44:49.554358 8547 master_runner.cc:186] Error getting master registration for 127.17.240.253:32831: OK, instance_id { permanent_uuid: "2648e93499a84392967f35adfc2478c2" instance_seqno: 1715010289363353 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240506 15:44:49.556565 8547 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 15:44:49.558034 8805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:44:49.558205 8805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:44:49.560309 8739 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [term 1 FOLLOWER]: Refusing update from remote peer b18d6697d12e4a0c80d25febc47c0c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:49.560602 8785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:49.561755 8771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18d6697d12e4a0c80d25febc47c0c2a. Latest consensus state: current_term: 1 leader_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } } }
I20240506 15:44:49.561839 8771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:49.563547 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.553109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.17.240.254:37467)
0506 15:44:49.553152 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.553153 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.555293 (+ 2140us) server_negotiation.cc:407] Connection header received
0506 15:44:49.555396 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.555397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.555408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.555435 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.556393 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.557041 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.557779 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.557974 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.559840 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.559843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.559844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.559850 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.561612 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.562446 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.562448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.562448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.562483 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.562752 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.562754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.562754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.562877 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.562957 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.563361 (+ 404us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.563453 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240506 15:44:49.563607 8785 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } } }
I20240506 15:44:49.563681 8785 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:49.564066 8771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } } }
I20240506 15:44:49.564149 8771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:49.564338 8808 mvcc.cc:205] Tried to move back new op lower bound from 7024682146036011008 to 7024682146013388800. Current Snapshot: MvccSnapshot[applied={T|T < 7024682146036011008}]
I20240506 15:44:49.564648 8809 mvcc.cc:205] Tried to move back new op lower bound from 7024682146036011008 to 7024682146013388800. Current Snapshot: MvccSnapshot[applied={T|T < 7024682146036011008}]
I20240506 15:44:49.566143 8805 catalog_manager.cc:1347] Generated new cluster ID: db1d16e6fe3f432799a2cd390eeb01fa
I20240506 15:44:49.566304 8805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:44:49.581627 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.554505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.17.240.252:45075)
0506 15:44:49.556640 (+ 2135us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.556641 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.562969 (+ 6328us) server_negotiation.cc:407] Connection header received
0506 15:44:49.564884 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.564886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.564896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.564925 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.567457 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.568097 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.569437 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.569603 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.577724 (+ 8121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.577728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.577728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.577746 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.580476 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.580860 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.580861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.580861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.580885 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.581035 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.581036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.581037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.581147 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.581231 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.581424 (+ 193us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.581522 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2113}
I20240506 15:44:49.582768 8805 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:44:49.583501 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.554037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.17.240.253:32831)
0506 15:44:49.554132 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.554133 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.562855 (+ 8722us) server_negotiation.cc:407] Connection header received
0506 15:44:49.564764 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.564765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.564779 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.564811 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.566645 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.567365 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.569861 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.570046 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.578013 (+ 7967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.578017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.578017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.578025 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.579765 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.582686 (+ 2921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.582687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.582688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.582717 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.582885 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.582885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.582886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.583002 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.583101 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.583311 (+ 210us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.583403 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20240506 15:44:49.583796 8805 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:44:49.595526 8690 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } attempt: 3
I20240506 15:44:49.596244 8805 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a: Generated new TSK 0
I20240506 15:44:49.596535 8805 catalog_manager.cc:1514] Initializing in-progress tserver states...
May 06 15:44:49 dist-test-slave-v0k1 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715010289, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.240.253@KRBTEST.COM for kudu/127.17.240.252@KRBTEST.COM
I20240506 15:44:49.607852 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.595834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.240.252:45075 (local address 127.0.0.1:36438)
0506 15:44:49.595919 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0506 15:44:49.595930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 15:44:49.595976 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 15:44:49.596266 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0506 15:44:49.596267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0506 15:44:49.596456 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0506 15:44:49.596658 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.596660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.597620 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.597622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0506 15:44:49.598189 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0506 15:44:49.598191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.598266 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.599507 (+ 1241us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0506 15:44:49.599524 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0506 15:44:49.601150 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0506 15:44:49.606529 (+ 5379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.606530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.606531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.606721 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.607048 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0506 15:44:49.607049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0506 15:44:49.607050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0506 15:44:49.607101 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0506 15:44:49.607515 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0506 15:44:49.607516 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0506 15:44:49.607577 (+ 61us) client_negotiation.cc:769] Sending connection context
0506 15:44:49.607635 (+ 58us) client_negotiation.cc:240] Negotiation successful
0506 15:44:49.607744 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20240506 15:44:49.608650 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.596062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.17.240.252:45075)
0506 15:44:49.596130 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.596132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.596139 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:49.596159 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.596160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.596170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.596214 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.596779 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.597535 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.601537 (+ 4002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.601751 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.601792 (+ 41us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0506 15:44:49.601796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0506 15:44:49.601796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0506 15:44:49.601800 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0506 15:44:49.606423 (+ 4623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.606827 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.606829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.606830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.606893 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0506 15:44:49.607180 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0506 15:44:49.607181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0506 15:44:49.607181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0506 15:44:49.607384 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0506 15:44:49.608242 (+ 858us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.608317 (+ 75us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.608498 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240506 15:44:49.610338 8690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2: Bootstrap starting.
I20240506 15:44:49.611119 8690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2: Neither blocks nor log segments found. Creating new log.
I20240506 15:44:49.611442 8690 log.cc:824] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2: Log is configured to *not* fsync() on all Append() calls
I20240506 15:44:49.612448 8690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2: No bootstrap required, opened a new log
I20240506 15:44:49.614229 8690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } }
I20240506 15:44:49.614367 8690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2648e93499a84392967f35adfc2478c2, State: Initialized, Role: FOLLOWER
I20240506 15:44:49.614437 8690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } }
I20240506 15:44:49.614725 8690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:44:49.614797 8664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2648e93499a84392967f35adfc2478c2"
I20240506 15:44:49.614903 8664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:44:49.615823 8664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b18d6697d12e4a0c80d25febc47c0c2a in term 1.
I20240506 15:44:49.616269 8663 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [term 1 FOLLOWER]: Refusing update from remote peer b18d6697d12e4a0c80d25febc47c0c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:44:49.614734 8816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } } }
I20240506 15:44:49.616586 8812 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:44:49.616675 8816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:49.617869 8816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18d6697d12e4a0c80d25febc47c0c2a. Latest consensus state: current_term: 1 leader_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } } }
I20240506 15:44:49.618000 8816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:49.618077 8816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } } }
I20240506 15:44:49.618168 8816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 15:44:49.619406 8828 mvcc.cc:205] Tried to move back new op lower bound from 7024682146179674112 to 7024682146013388800. Current Snapshot: MvccSnapshot[applied={T|T < 7024682146036011008}]
I20240506 15:44:49.621474 8827 catalog_manager.cc:1259] Loaded cluster ID: db1d16e6fe3f432799a2cd390eeb01fa
I20240506 15:44:49.621527 8827 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2: loading cluster ID for follower catalog manager: success
I20240506 15:44:49.622840 8827 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2: acquiring CA information for follower catalog manager: success
I20240506 15:44:49.623225 8827 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2648e93499a84392967f35adfc2478c2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240506 15:44:49.629674 8618 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.628081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.17.240.254:37467)
0506 15:44:49.628124 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.628126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.629035 (+ 909us) server_negotiation.cc:407] Connection header received
0506 15:44:49.629136 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.629137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.629147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.629176 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.629603 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38940: BlockingRecv error: recv got EOF from 127.0.0.1:38940 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20240506 15:44:49.630386 8694 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.628455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.17.240.253:32831)
0506 15:44:49.628489 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.628491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.629907 (+ 1416us) server_negotiation.cc:407] Connection header received
0506 15:44:49.630002 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.630003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.630012 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.630040 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.630338 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58888: BlockingRecv error: recv got EOF from 127.0.0.1:58888 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20240506 15:44:49.631088 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.628748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.17.240.252:45075)
0506 15:44:49.628785 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.628787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.630587 (+ 1800us) server_negotiation.cc:407] Connection header received
0506 15:44:49.630678 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.630679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.630689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.630716 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.631030 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36450: BlockingRecv error: recv got EOF from 127.0.0.1:36450 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20240506 15:44:49.639228 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.635786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.17.240.253:32831)
0506 15:44:49.635833 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.635835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.636151 (+ 316us) server_negotiation.cc:407] Connection header received
0506 15:44:49.636284 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.636284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.636310 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.636363 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:49.636836 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.637608 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.638529 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.638700 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.638835 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:49.638943 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:49.638993 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.639130 (+ 137us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.639152 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20240506 15:44:49.642539 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.641681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.17.240.252:45075)
0506 15:44:49.641718 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.641720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.641962 (+ 242us) server_negotiation.cc:407] Connection header received
0506 15:44:49.642075 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.642076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.642096 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.642127 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.642483 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36466: BlockingRecv error: recv got EOF from 127.0.0.1:36466 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:44:49.647567 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.644771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.17.240.253:32831)
0506 15:44:49.644805 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.644807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.645038 (+ 231us) server_negotiation.cc:407] Connection header received
0506 15:44:49.645129 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.645129 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.645139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.645165 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:49.645510 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.646216 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.647071 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.647240 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.647340 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:49.647367 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:49.647392 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.647481 (+ 89us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.647498 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20240506 15:44:49.650578 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.649793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.17.240.252:45075)
0506 15:44:49.649843 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.649844 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.650072 (+ 228us) server_negotiation.cc:407] Connection header received
0506 15:44:49.650167 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.650168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.650179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.650206 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.650526 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36468: BlockingRecv error: recv got EOF from 127.0.0.1:36468 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20240506 15:44:49.656571 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.653707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.17.240.253:32831)
0506 15:44:49.653742 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.653744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.653969 (+ 225us) server_negotiation.cc:407] Connection header received
0506 15:44:49.654064 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.654065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.654075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.654102 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:49.654454 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.655215 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.656054 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.656213 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.656311 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:49.656358 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:49.656385 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.656483 (+ 98us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.656502 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20240506 15:44:49.659592 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.658821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.17.240.252:45075)
0506 15:44:49.658858 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.658860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.659073 (+ 213us) server_negotiation.cc:407] Connection header received
0506 15:44:49.659164 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.659165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.659175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.659202 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.659543 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36476: BlockingRecv error: recv got EOF from 127.0.0.1:36476 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:44:49.667577 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.664741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.17.240.253:32831)
0506 15:44:49.664779 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.664781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.665013 (+ 232us) server_negotiation.cc:407] Connection header received
0506 15:44:49.665103 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.665104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.665116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.665147 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:49.665485 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.666229 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.667058 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.667217 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.667329 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:49.667358 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:49.667386 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.667485 (+ 99us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.667504 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20240506 15:44:49.670581 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.669798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.17.240.252:45075)
0506 15:44:49.669848 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.669850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.670078 (+ 228us) server_negotiation.cc:407] Connection header received
0506 15:44:49.670165 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.670166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.670177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.670202 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.670531 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36482: BlockingRecv error: recv got EOF from 127.0.0.1:36482 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20240506 15:44:49.682996 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.679963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.17.240.253:32831)
0506 15:44:49.680007 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.680009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.680292 (+ 283us) server_negotiation.cc:407] Connection header received
0506 15:44:49.680406 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.680407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.680419 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.680447 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:49.680836 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.681555 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.682398 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.682582 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.682705 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:49.682737 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:49.682766 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.682887 (+ 121us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.682912 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20240506 15:44:49.686606 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.685601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.17.240.252:45075)
0506 15:44:49.685640 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.685642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.685995 (+ 353us) server_negotiation.cc:407] Connection header received
0506 15:44:49.686096 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.686097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.686109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.686137 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.686549 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36490: BlockingRecv error: recv got EOF from 127.0.0.1:36490 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20240506 15:44:49.707273 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.704131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58944 (local address 127.17.240.253:32831)
0506 15:44:49.704177 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.704179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.704473 (+ 294us) server_negotiation.cc:407] Connection header received
0506 15:44:49.704581 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.704582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.704593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.704621 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:49.705026 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.705730 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.706627 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.706798 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.706933 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:49.706965 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:49.706993 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.707164 (+ 171us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.707188 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20240506 15:44:49.710563 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.709707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.17.240.252:45075)
0506 15:44:49.709753 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.709755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.710014 (+ 259us) server_negotiation.cc:407] Connection header received
0506 15:44:49.710112 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.710112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.710124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.710152 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.710508 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36504: BlockingRecv error: recv got EOF from 127.0.0.1:36504 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20240506 15:44:49.746973 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.744073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.17.240.253:32831)
0506 15:44:49.744119 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.744121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.744394 (+ 273us) server_negotiation.cc:407] Connection header received
0506 15:44:49.744492 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.744492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.744503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.744530 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:49.744898 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.745628 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.746445 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.746608 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.746706 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:49.746734 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:49.746760 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.746873 (+ 113us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.746891 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20240506 15:44:49.750129 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.749378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.17.240.252:45075)
0506 15:44:49.749418 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.749419 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.749615 (+ 196us) server_negotiation.cc:407] Connection header received
0506 15:44:49.749702 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.749703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.749713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.749739 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.750069 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36510: BlockingRecv error: recv got EOF from 127.0.0.1:36510 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:44:49.817952 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.815693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.17.240.253:32831)
0506 15:44:49.815734 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.815736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.815747 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:44:49.815836 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.815837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.815845 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.815892 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:49.816234 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.816755 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.817610 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.817712 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.817727 (+ 15us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:49.817747 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:49.817784 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.817873 (+ 89us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.817888 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20240506 15:44:49.820323 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.819711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.17.240.252:45075)
0506 15:44:49.819762 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.819764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.819844 (+ 80us) server_negotiation.cc:407] Connection header received
0506 15:44:49.819910 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.819911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.819922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.819977 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.820265 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36520: BlockingRecv error: recv got EOF from 127.0.0.1:36520 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20240506 15:44:49.951582 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:49.949409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.17.240.253:32831)
0506 15:44:49.949478 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.949480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.949644 (+ 164us) server_negotiation.cc:407] Connection header received
0506 15:44:49.949723 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.949724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.949735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.949808 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:49.950076 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.950566 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:49.951159 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:49.951311 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:49.951328 (+ 17us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:49.951351 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:49.951401 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:49.951471 (+ 70us) server_negotiation.cc:299] Negotiation successful
0506 15:44:49.951491 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20240506 15:44:49.953720 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:49.953212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.17.240.252:45075)
0506 15:44:49.953281 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:49.953283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:49.953293 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:44:49.953351 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:49.953352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:49.953362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:49.953411 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:49.953672 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36536: BlockingRecv error: recv got EOF from 127.0.0.1:36536 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20240506 15:44:50.102421 8812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18d6697d12e4a0c80d25febc47c0c2a" member_type: VOTER last_known_addr { host: "127.17.240.254" port: 37467 } } peers { permanent_uuid: "2648e93499a84392967f35adfc2478c2" member_type: VOTER last_known_addr { host: "127.17.240.253" port: 32831 } } peers { permanent_uuid: "032bb8b800c641ecb2d62ae84de1ffea" member_type: VOTER last_known_addr { host: "127.17.240.252" port: 45075 } } }
I20240506 15:44:50.102622 8812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18d6697d12e4a0c80d25febc47c0c2a [sys.catalog]: This master's current role is: LEADER
I20240506 15:44:50.213060 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:50.211044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.17.240.253:32831)
0506 15:44:50.211111 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:50.211113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:50.211124 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:44:50.211203 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:50.211204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:50.211218 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:50.211268 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:50.211574 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.212051 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:50.212628 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.212770 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:50.212786 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:50.212808 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:50.212857 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:50.212933 (+ 76us) server_negotiation.cc:299] Negotiation successful
0506 15:44:50.212958 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20240506 15:44:50.215358 8768 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 15:44:50.214801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.17.240.252:45075)
0506 15:44:50.214861 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:50.214863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:50.214873 (+ 10us) server_negotiation.cc:407] Connection header received
0506 15:44:50.214898 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:50.214899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:50.214908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:50.214964 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0506 15:44:50.215312 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36548: BlockingRecv error: recv got EOF from 127.0.0.1:36548 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20240506 15:44:50.538271 8782 catalog_manager.cc:1259] Loaded cluster ID: db1d16e6fe3f432799a2cd390eeb01fa
I20240506 15:44:50.538353 8782 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea: loading cluster ID for follower catalog manager: success
I20240506 15:44:50.539228 8782 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea: acquiring CA information for follower catalog manager: success
I20240506 15:44:50.539484 8782 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 032bb8b800c641ecb2d62ae84de1ffea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 15:44:50.730592 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:50.728435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.17.240.253:32831)
0506 15:44:50.728548 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:50.728549 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:50.728560 (+ 11us) server_negotiation.cc:407] Connection header received
0506 15:44:50.728635 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:50.728636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:50.728647 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:50.728707 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:50.728953 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.729439 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:50.730108 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.730219 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:50.730234 (+ 15us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:50.730255 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:50.730298 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:50.730425 (+ 127us) server_negotiation.cc:299] Negotiation successful
0506 15:44:50.730472 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20240506 15:44:50.734326 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:50.732383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.17.240.252:45075)
0506 15:44:50.732525 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:50.732527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:50.732535 (+ 8us) server_negotiation.cc:407] Connection header received
0506 15:44:50.732559 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:50.732559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:50.732601 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:50.732676 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:50.732881 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.733363 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:50.733875 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.733994 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:50.734040 (+ 46us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:50.734126 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:50.734186 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:50.734239 (+ 53us) server_negotiation.cc:299] Negotiation successful
0506 15:44:50.734263 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20240506 15:44:50.738605 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:50.736264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.17.240.252:45075)
0506 15:44:50.736299 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:50.736301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:50.736308 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:50.736359 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:50.736360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:50.736372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:50.736419 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:50.736745 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.737476 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:50.738126 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.738291 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:50.738322 (+ 31us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:50.738349 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:50.738405 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:50.738482 (+ 77us) server_negotiation.cc:299] Negotiation successful
0506 15:44:50.738513 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240506 15:44:50.738729 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:50.736253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.17.240.253:32831)
0506 15:44:50.736294 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:50.736296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:50.736303 (+ 7us) server_negotiation.cc:407] Connection header received
0506 15:44:50.736327 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:50.736327 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:50.736341 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:50.736387 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:50.736708 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.737477 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:50.738248 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.738431 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:50.738451 (+ 20us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:50.738478 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:50.738524 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:50.738620 (+ 96us) server_negotiation.cc:299] Negotiation successful
0506 15:44:50.738650 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240506 15:44:50.739145 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0506 15:44:50.735967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.17.240.254:37467)
0506 15:44:50.736064 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0506 15:44:50.736066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0506 15:44:50.736082 (+ 16us) server_negotiation.cc:407] Connection header received
0506 15:44:50.736167 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0506 15:44:50.736168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0506 15:44:50.736199 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0506 15:44:50.736358 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0506 15:44:50.736661 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.737635 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0506 15:44:50.738347 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0506 15:44:50.738533 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0506 15:44:50.738602 (+ 69us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0506 15:44:50.738699 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0506 15:44:50.738989 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0506 15:44:50.739031 (+ 42us) server_negotiation.cc:299] Negotiation successful
0506 15:44:50.739069 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":45,"threads_started":1}
I20240506 15:44:50.740770 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 8547
I20240506 15:44:50.746038 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 8622
I20240506 15:44:50.751194 18371 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb5AjvH/build/release/bin/kudu with pid 8697
2024-05-06T15:44:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2755 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5493 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (311992 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
1 FAILED TEST